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Achievements - OPMB Analytics l 


v integrated Workload Distribution Project (IWD) aiia 


> s.16(2) 
> N/R 


> N/R 


¥ Electronic Travel Authorization 
> N/R 


{Express Entry, phase |: 
> N/R 
> N/R 
v Validated and contributed to OMC-PID’s risk profiles on different LOBs 


¥ Ad-hoc requests and other exploratory projects: 
> N/R 


¥ Contributed to Data Governance: 


N/R 
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Some notes on number of cases needed for testing or revision of 


For Express Entry, DPAT should be able to conduct the first testing and if necessary revision of the model 
when actual FinDec will be available for at least 3,000 applications. This was estimated by conducting 
several trials on randomly taken sample of 2% (2,936) of FSW data set (the most challenging to model) 
and the results were proven to be reasonable and valid. The monitoring of Express Entry admissions on 
weekly basis is necessary to establish the time period for required actual data be available. 


As for March 15, 2015, CIC has received 737 applications from invited candidates (about 21.5% of 


invitations). 


Taking into the consideration this recent trend in admissions, DPAT believes that testing and revision of 
the model have to wait several months with possibility to wait to the end of 2015. 


The following is the presentation of results for FSW model for 2% of data that is 2,936 applications. 


There were five trials conducted to see if the model still results in accurate prediction for five randomly 
selected 2% samples of data set. 


Trial i 
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Trial 3 
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Trial 4 


The comparison of predicted FinDec with Actual FinDec 
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Trials Results 


The accuracy of the model varied insignificant from 86% to 89% for five randomly selected data samples. 
This indicates the robustness of the model for 2% sample of the data. 


Note The new model will be built using more fields and historical information. This will increase 
accuracy of the model as it was observed from the work done for other LoBs. 
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Express Entry- Data Exploratio 


Original data set had 547,215 applications with 122 unique fields 


> Federal Skilled Worker Program (FSW)- - 72.13% 
>  Provincial/Territorial Nominees (P/N) - 21.16% 
> Canadian Experience Class (CEC) - 6.69% 

>  Skilled Trades (ST) - 0.03% 


Distribution by Programs 


galed Workers 
=rovincia /Terrtorial Nominees © 


Cenacian Experience Class 
Skilled Trades | 


e Missing values and outliers were removed from the data for each LOB 

e As a result, each LOB has reduced number of records 

e Time periods for modeling were also chosen for each LOB depending on availability 
and quality of the data 
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e Total number of valid records consist of 155,665 applications 
e Total number of fields is 122 unique fields 
e Dates covered (final decision dates) are from 2007-01-01 to 2008-02-27 


Total number of fields is 122 unique fields 
Dates covered (final decision dates) are from 2007-01-02 to 2014-08-18 


Total number of fields is 122 unique fields 
| © Dates covered (final decision dates) are from 2011-07-16 to 2014-08-15 
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e  DPAT used Auto data Preparation method 
to identify the most important predictors 
for making decision to approve or refuse 
the application. 


e  DPAT have looked at different prediction 
methods, such as Decision Trees (i.e. 
C5.0, CHIAD...) and Logistic Regression. 


e After developing models based on each 
of these methodologies, we compare the 
overall accuracy rate, discuss the results 
with the clients and select the method 
with the highest possible accuracy rate 
that meets CIC’s needs. 


Citizenship and Citoyenneté et Can a A 5 
immigration Canada Immigration Canada A-2018-09320-000113 


Pages 114 to / à 116 
are withheld pursuant to sections 


sont retenues en vertu des articles 


16(2), 16(1)(b) 


of the Access to Information Act 


de la Loi sur l'accès à l'information 


Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


Citizenship and Citoyenneté et Car a A 9 
immigration Canada immigration Canada A-2018-09320-000117 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


Strategic Business Research Projects 


Operations Planning and Performance Branch 
Business Research, intelligence and Analytics Directorate 
February 24, 2017 


$ & Cuers ee CRP ei 
rapain Carada  hewsignsion Cant 


A-2018-09320-000118 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


Background 


OPP won two Request for Proposals issued by the Canadian Safety and Security Program (CSSP} to 
undertake collaborative innovation projects entitled FASTER PrivBio and Chain of Trust. 


+ FASTER PrivBio combines the efforts of IRCC, Canada Border Services Agency, WorldReach Software 
Corporation, University of Ottawa and Ryerson University to develop the prototype of a smartphone 
application (App) that enables advance, remote, secure and privacy preserving pre-screening of 
applicants before they arrive at the physical border. Approved applicants can use the App ta self- 
authenticate and provide travel authorization using the App at self-service border kiosks. 


*  Rurining on-time and on-budget. Project lifespan from August 2015 to March 2017. Budget of $2.3M; 
IREE in-kind investment of $126K; receiving $78K in CSSP funding. 


Building on the successes af FASTER PrivBio, Chain of Trust combines the efforts of IRCC, Canada 
Border Services Agency, WorldReach Software Corporation, Entrust Datacard, Face4 Systems and 
University of Ottawa to build automated adjudication rules to continuously assess risks and triage 
applicants and travellers using the powers of predictive analytics. 


¢ Project will run from April 2017 to March 2018. Budget of $2.8M: IRCC in-kind investment of S188K: 
receiving 5237K in CSSP funding. 


What is the Canadian Safety and Security Program (CSSP}? 


The CSSP is a federally-funded competitive granting program administered by Defence Research 
and Development Canada {[DRDC]. The program funds government-led R&D projects in 
collaboration with response and emergency management organizations, non-governmental 

\ agencies, industry, and academia. 


Defence Research and Development Canada’s Canadian Safety and Security Program regularly 
issues RFP’s in search of Science and Technology projects that strengthen Canada’s ability to 
anticipate, prevent, mitigate, prepare for, respond to, and recover from serious threats or 
incidents to Canada. 


IRCC has been successful in winning two RFPs, receiving funding to innovate IRCC’s program 
application processes, while increasing client convenience, program integrity and national 
security. 


The first project, entitled FASTER Privio combined the efforts of IRCC, CBSA, two universities and 
one private company. As planned, the project build the prototype of a mobile application, with 
eTA as the use case, that enabled the advance, remote, secure and private collection of 
information to enhance the screening of eTA applicants. Approved applicants receive a private 
and secure token stored on their smartphone to clear border processes. The prototype has 
been delivered on-time and on-budget, with project coming to an end in March. 


The second project, entitled Chain of Trust, combines the efforts of IRCC, CBSA, a university and 
three private companies, builds on the successes of FASTER PrivBio to build automated 
adjudication rules to continuously assess risks and triage applicants and travellers using the 
powers of predictive analytics. Project will start in April 2017 and will run for one year. 


Projects are prototypes. They do not use real data, nor are they linked to any GoC systems. 
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FASTER-PrivBio Project - Benefits 


Using the Electronic Travel Authorization Program (eTA) as a use-case, developed a smartphone App 
prototype that provides an innovative, mobile, end-to-end eTA application solution that accelerates 
processes for low-risk travellers and triages applications for manual adjudication (at time of application 
and at the self-service border kiosk). 
« — For the traveller 

© Convenience of using e smartphone, from time of application, to time of entry into Canada. 


+ Forthe Get 
« Potential increase of public safety and national security: 
* Stronger applicant identity by early verification of ePassport and facial biometrics 


+  Reallocetion of resources to higher risks applicants and travellers 


» Privacy of personal information is built in the architecture. 


Although the prototype was build using the eTA, several IRCC programs could benefit 
from the concept. 


FASTER PrivBio uses technologies that already exist and, that are commonly used by 
consumers, businesses and governments around the world. Namely the smartphone 
and its camera. 


The project takes these technologies and combines them together to build an 
application that simplifies and accelerates the processing of low-risk travellers and 


triages applications for manual review, whether at time of application or at the border. 


For the traveller, the solution brings added convenience. Use of smartphones are 
widespread and keep on increasing in numbers and possibilities. 


For the Government of Canada, the solution increases efficiencies, public safety and 
national security. 
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FASTER-PrivBio Project 
FASTER Component 


Prototype shows that the secure remote collection and verification of an applicant's biometrics and biographical data can 
be done at pre-travel stage. Smartphone capabilities can enhance risk based screening of immigration applications. While 
the Project focused on the aTA as the use-casa, other IRCC programs could benefit trom the capabilities and maybe extend 
lower risk programeligibilityto more travellers. 


. Design OE SMARTPHONE APP PROTOTYPE 
« User friendly interlace (drop down menus, visual aids, minimal data capture, etc) 
~  Bonefitsof smartphone Near Feld Communication and Camera capabilities 
BOVEY QE PERSONAL PRT ORMATION HUUT INTO THE ADR ARCHITECTURE 
- Sestorage of pearsanal datacn mobile device 
~ Secure transmission of personal data to servers 


CAPTURE Ge EPASSPONT'S BOMETRR Re MOGRAPHH. DATA ENSURES DATA INTEGRITY 


STATUS NOTIHCATONS AVALARLE ON MOBLE APP 


SMARTEHONE Quick Response CODE ie CONTAINING TRAVELLER INFORMATION ANG ETA AUTHORIZATION TO BE USED AT BORDER SELF- 
LE ROSKS 


OK, now how does this work? Well, basically, the solution has two components: 


1. 
2. 


The application process (FASTER) and 2. 
The Validation process at the border kiosk (PrivBio) 


As it relates to the application process, the applicant: 


Downloads the APP to the smartphone from Google Store (Android based only for now) 
Creates his eTA Account 

Takes a picture of his/her ePassport Datapage 

Places the smartphone on the ePassport to transfert content of the chip to smartphone 
Takes a selfie 

Completes the eTA application by answering Program questions and make payment 


* Information is packaged and securily transmitted to IRCC servers. 


* The application is very easy to use, having gone through usability testing 

* The protection of personal information was built into the solution. The smarthone does not 
store personal information, and the security of the transmission are at the forefront of 
technologies. 

e Applicant biographic and biometric information is pulled from the passport chip, so limited 
chances of errors ( program integrity) and keying is all done by applicant 
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FASTER PrivBio Project 
PrivBio Component 


Prototype shows that the secure and remote storage of an individual's biometrics and biographical 
data and travel approval can be done at pre-travel stage. The private information stored on the 
smartphone can be used to verify a traveller's identity and travel authorizations at border self- 
service kiosks. 


* PROOF OF CONCEPT ~ PRIVACY PRESERVING BIOMETRIC IDENTITY ALGORITHM 
No facial templates on the smartphone 
o Biometrics cannot be reverse engineered 
c Encryption accounts for biometric variability 


* PROIECT Proves THAT THE VISION OF THE FUTURE IS POSSIBLE 
~ Mobile device can become the travel document 
+ At tine of application remote use af travel document & biometric 
* At border self-service kiask: privacy preserving biometric k authorization 
Physical book {ePassporti remains a needed companion 
— As itstands, the Smartphone App could complement existing processes for a meaningful segment of 
international travellers to enhance facilitation and security 


PrivBio is the second component of the solution. Basically, the component uses 
complex mathematical formulas to: store personal information, the facial biometric, the 
eTA number and passport information on the smartphone in the form in the form of a 
key. The smartphone and the ePassport is used at the border self-service kiosk. 


So how does it work. At the border self-service kiosk, 

e the applicant will insert his passport in a reader and tap his smartphone against 
another reader (similar to « tap and pay » at retailers such as Starbucks) 

e Kiosk will unlock content of chip and telephone 

e The kiosk will take a picture of the traveller and compare it to the ePassport and 
smartphone 


Protection of private information was key throughout the project 
e No personal information is stored on the phone 
e Facial biometric cannot be drawn from phone 


Solution proves that convenience, privacy and security can exist together ona 
smartphone device. 
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Chain of Trust Project 
Benefits 
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* FOR THE TRAVELLER 
+ Faster processing and improved client experience for low-risk travellers 


© FOR PROGRAM INTEGRITY 


» Increased likelihood that high-risk individuals will be flagged for scrutiny 


* For innovanon ar IRCC 
» much-needed funding to advance predictive analytics 
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Chain Of Trust Project 
Components 


Building on the successes of FASTER PrivBio, the Chain of Trust Project aims to enhance the prototype to: 


1, COLLECT ADDITIONAL DATA EARLY IN THE TRAVEL CONTINUUM, INCLUDING! 
+ Enable online submission of the E311 Declaration Form prior to arrival in Canada. 
CONSOUDATING ALL DATA COLLECTED ABOUT THE TRAVELLER THROUGHOUT THE TRAVEL CONTINUUM 
DEVELOP A COLLABORATIVE PREDICTIVE ANALYTICS RISK ASSESSMENT BY IRCC AND THE CBSA 
+ Each agency will develop their own predictive models to assess the risk posed by travellers. 


IRCC's risk assessment during eTA (use case) adjudication will be shared with the CBSA and 
incorporated into their risk assessment. 


CBSA will update its risk assessment as each traveller advances through the travel 
continuum. 


OPPB will focus an objective 3 


Additional info upon check-in. 
Additional info when Declaration Form filled. 
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Next Steps 


+ FASTER PrivBio PROJECT 
— Present End-to-End Concept to CSSP Program Director 
~ Complete Close-Out Reports 


— Suggest the App for pilot consideration within an [RCC Program 


* CHAIN OF TRUST Project 
Work will occur during 2017-18 
Project charter is finalized and currently in the approval process 
Procurement activities with private-sector partners to be completed by April 1, 2017 


» OPP will receive $350K from the CSSP to manage the contract with WorldReach Software 
Corporation, responsible for the improvements to the smartphone App 


» The CBSA will manage contracts with the other partners 


OPP will continue to work closely with the IPG and {RG during the development of risk 
indicators 


A demonstration of the prototype is planned for early 2018 
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Data and Predictive Analytics: Opportunities for CIC 


What is Predictive Analytics and why do we do it? 


The term “predictive analytics” describes the process of discovering patterns in past behavior to predict 
the outcome of future events. It combines computer science, statistics and operations research to drive 
smarter decisions by extracting actionable insights from the vast quantities of data captured by 
government departments. 


Predictive analytics helps prevent fraud and reduce waste. 
Aside from the operational efficiencies gained, predictive 
analytics can be used to standardize decision making processes, increasing fairness and transparency. 


Predictive analytics will reduce the labour intensity of application triaging and processing as well as 
minimize the time to a decision for a significant number of cases. 


Projects under way (see below for all the details): 


Predictive Analytics moves forward CIC’s modernization agenda by supporting key initiatives such as 
Integrated Network Project, Electronic Travel Authorization (eTA), and Express Entry. 


Progress to Date: 


1- MONEY and People for Predictive Analytics: 


N/R 
We have money that will be temporary (some of Express 
Entry for example) and money that will be ongoing, for delivering projects and to establish an ongoing 
function in CIC so that models can be fully used by regular monitoring and improvement. 


We have therefore a bit over 800,000S? for each of the first few years to establish the team and the 
function, and the ongoing funding is around 700,000S?. Of note though that the Integrated Network 
project is currently expanding beyond TR, and DPAT did not receive funding for work beyond TR and a 
few PR lines. The project and ongoing funding might change in the near future. The team was officially 
formed in March 2014, first with two people, now with 6.5 full time resources. 


2- Projects we are working on and progress to date: (see the attachment for a full picture) 


1 For the current projects 
2 Of note, the USA as well as other countries, has a budget of millions for the similar teams and functionalities. 


Paper by DPAT (MJD and IB), Feb 10°", 2015. 1 
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- Models for Automated triaging and for auto-assignment of Express Entry applications by 
December 2014 for phase 1 and for the Fall 2015 for Phase 2 of modeling (to rebuild with 
real express entry data), for a roll out of a Predictive Analytics model in late spring 2016; 


What that means is that we have work and models to develop until Spring 2018 for now, (pending IN 
Project confirmation of dates) and ongoing monitoring and maintenance of potentially up to 30 models 
(according LOBs). 


Engagement and IM/IT: 


Predictive Analytics Working Groups: 
DPAT is holding bi-weekly meetings for each of the ongoing projects. The purpose of these working 
groups is to combine operational business knowledge and expertise with statistical analysis. 


Engagement of other departments and other countries: 


Predictive analytics is a fast moving technology and maintaining external partnerships is important to 
ensure that business objectives are met and that technology focused activities are aligned with the 
broader CIC modernization. We, OPMB-PMU-DPAT initiated and chair an Analytics Coalition, in a 
collaborative effort with 13 other government departments and the Privy Council’s Central Innovation 
Hub. We have also engaged in conversation with the UK (by CBSA LIO), 
with the Australian (directly); and we visited the US last year. 


Engagement and working with SIMB / IT: 


Paper by DPAT (MJD and IB), Feb 10°", 2015. 2 
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Functionality for changing business rules (the models) needs to be independent of GCMS releases, and 
be centrally modified by the program in order to control program risk or in order to manage workload. 
Just-in-time quality assurance will also be made available. 


Opportunities for Success: 
Decision making processes at CIC can be enhanced by using predictive analytics models, which offer the 
following benefits: 


e Replaces rules and risk profiles based on subjective input and “gut” feelings with evidence 
based, data driven business rules. 

e Increases operational efficiency by reducing the number of paper based, manual business 
processes. 

e Improves the transparency and consistency of decisions across the CIC processing network. 

e Decreases the effort required to do quality assurance. 


e Enables triaging and organizing application workload for optimal throughput. 
e Can be used as legal evidence to support CIC. 


Success is measured in terms of the business value delivered: faster application processing, consistent 
decisions and optimized business processes. Business user satisfaction and an improved client 
experience are also key success factors. Long term success will be evidenced by a pervasive culture of 
analytical thinking at CIC, mathematically sound practices, and the capacity to participate with other 
government departments on predictive analytics projects. Key aspects of successful projects as it relates 
to Data and Predictive Analytics: 


Simplify the concept of predictive analytics for business users and managers. 

A consistent process for collaborating with the business stakeholders 

Specific business outcomes for predictive analytics work 

Adequate IT support 

Proper communication of the results of predictive modeling to business stakeholders 


OW ee, Be 


Having timely insights from predictive models in front of the business staff that can best use 
them 

7. Aclosed-loop feedback process for improving and updating predictive models. 

8. Executive endorsement of the results and for data driven decision making 


Predictive analytics projects have seven main phases: 
1. Collection of Data & Business Requirements 

Analysis of Preliminary Data 

Development of Predictive Models 

Validation of Predictive Models 

Statistical modeling of risk/complexity scoring 

Implementation of Predictive Models 


DE Ors OS 


Monitoring and Maintenance of Models 


Paper by DPAT {MJD and IB), Feb 10°", 2015. 3 
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SK or Hubert: 
Predictive analytics have proven value in private industry and government 


Predictive Analytics are a key piece of the modernization strategy 
DPAT is working closely with partners: SPO, PID and OMC 


CIC is at the very beginning of using predictive analytics 
We are catching up on other countries (USA and Australia for example) and other 
departments 


Expectations need to be set carefully 


Predictive analytics can be transformative for an organization, 
but it takes a long time to put everything in place 


1 
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L'analyse prédictive s CIC 


‘Modélisation de fanalyse prédictive pour Entrée express 


Entrée express ~ Exploration des dunndes 


Prochaines stapes 


Annexes ii: Lecans tirées des autres ministires et gouverterents Grangers 
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# L'analyse prédictive est le processus consistant à dégager les tendances dans les 
données et comportements historiques pour inférer la probabilité d'événements 
futurs et prévoir les résultats probabiles., 


» L'analyse prédictive permet d'utiliser les données pour orienter les décisions 


Mise sur une combinaison de... Avantages 
Sciences informatiques Confirme forme 
Statistiques 
Expertise opérationnel 
Logiciels spécialisée 


sirgs at, 
Ee ES 


MJ opens and explains definition and how it works. Also talk about concrete benefits in any 
business doing predictive analytics (Walmart, for example, or Google, that both use that 
now in their business to save money, profile their consumers and improve resource 
allocations) 


The benefits of combining operations expertise with empirical data and statistical analysis 
to build a predictive model: 


*increasing processing capacity 


*Facilitate program integrity exercises 
«Deliver optimal client service 


9 
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» L'analyse prédictive est fondée sur analyse statistique et la modélisation 
prédictive 
*» Appie les initiatives clés, p.: , la gestion intégrée de la 
charge de travail, Autorisation de voyage électronique [AVE] et Entrée express. 


Facilite ot appuie lo programme de modernisation 
S'ajoute aux outils de CIC pour déceler ja fraude 


yae fes 
données 


Les données sont 
saisies. mals ne 
soni pat 
mem mises 
t & thes ting 
apérationnelias 
Creer des poeskes 
Sotiris wy meios 
garata 4 it CHOSE ef di 
Patemon pink razede 


Read the deck bullets and insist on OUTCOMES: 
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quip é alyse prédictive [EDAP] 
Nouvelle équipe au sein de la DGGRO/UGR 


ti 


Passeport: 


æ Masson 


» L'analyse prédictive nécessite un engagement à long terme 


we Fe 


Team is new. 

When did we (CIC) get into predictive Analytics: 

TR MC and Express Entry, this year (2014) were the first projects that got approval to do 
predictive Analytics. 

TRMC: 500K app in FTE for 

Express Entry : 450K in O&M and 3 FTEs 

ETA : 2 FTEs 


Insist on Predictive Analytics capacity building will not happen overnight. Predictive 
Analytics take time to implement. 

OTHER COUNTRIES and Other Departments have done this for a few years now: proven 
value to achieve better allocation of resources. 


Building partnerships is key; 


Passport is certainly a key one we need to work with. 

And We say CIC here but a few words on passport: 

-Passport has been doing long term forecasting for years in order to forecast volume of 
work by office; and that is related to the funding model of passport, i-e revolving funds. 
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Racucily tes clonnden et Habis es exigances 
operatouneligs 


Efece ade analyse prdéininnin des données 
Concave les madéles pradictife 
Yalider tes modéles édités 


Procéder a la modi hestion statiique des cotes lére 


au (equela la cee iit 
ee 


-o 


Marie-Josée: 


-Partners and clients: 
-OPS partners, such as OPMB-PMU, OMC-PRPD, OMC-PID, IR, 
CPR, and SPO. 


-Historical data: 


FSW: Total number of valid records consist of 155,665 applications 


Total number of fields is 122 unique fields 
Dates covered (final decision dates) are from 2007-01-01 to 2008-02-27 


«Total number of valid records consist of 111,060 applications 


Total number of fields is 122 unique fields 
Dates covered (final decision dates) are from 2007-01-02 to 2014-08-18 


Cc 
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CEC: Total number of valid records consist of 25,058 applications (GCMS only) 
Total number of fields is 122 unique fields 
Dates covered (final decision dates) are from 2011-07-16 to 2014-08-15 


Concrete benefits in any business doing predictive analytics (Walmart, for example, or Googles, that 


The benefits of combining operations expertise with empirical data and statistical analysis to build a 
predictive model: 


eIncreasing processing capacity 


e Facilitate program integrity exercises 
e Deliver optimal client service 


Data Analytics: out reach 


Data Analytics Working Groups 
predictive analytics demands tight collaboration with partners and the business 
We started the WG in collaboration with PID. We co-chair. 


Data Analytics Coalition 

Great forum for sharing ideas and best practices as predictive analytics is relatively new to the federal 
government 

Similar goals — workload distribution 

Common challenges — change management, setting users expectations, poor data quality 
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+ Le nombre total de dossiers valides représente 156 666 demandes 
+ Le nombre total de champs esi 122 domaines uniques 
«lus dates couvertes {date do la décmwn finale} vont du 04-04-2007 au 27022008 


Le nombre total de dossiers valides représente 111 060 demandes 
«Le nombre total de champs est 122 domaines uniques 
+ Les dates couvertes (date de la décision finale} vont du 0201-2007 au 18082014 


-Le nombre total de dossiers valides représente 26 058 demandes (SMGC seulement) 
“le nombre total de champs est 122 domaines ungues 
«fs dates couvertes {date de la déciminn finale) vont du 16072041 au 48082014 


For FSW, DPAT analyzed data for FSW before and after MI. As a result, before MI data was 
chosen for FSW. This decision was based on business knowledge and opinion on which 
cases should be included in the sample in the way that there would not be any lost in 
representativeness of our future population of applications, while preserving the quality of 
the data set. The fact that Express Entry does not have MI restrictions is also consistent 
with this choice of pre-MI time period for modeling. 


For PN, data was analyzed for GCMS and CAIPS separately due to some missing values 
problem. since both sets of 
data yield the same acceptance rate and adding more applications to build the model 
results in a higher degree of freedom leading to increased robustness. Thus helping to 
ensure that the model performs as expected (i.e. is not over-fitted). 


For CEC, WG suggested that DPAT use only GCMS data from 2011 to 2014-07-04 (final 
decision date) given all the missing values in CAIPS. 


3 
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On appliquera des régles opérationnelles 8 chaque demande d'Entrée express reçue via le SMGC, 
puis celesti cdiendeont seront un score (pourcentage) selon le modéle et les indicateurs 
prédictifs, conformément au made élabors à faide des données d'Émréc express Qo modèle 
provisoire sera validé à l'aide des données « réelles » tirées des données d'Enirée express pour une 
pésode de 6 à 9 mas) 


Une fois un score (pourcentage) attribué à la demande, le sysiene assiqnera automatiquement to 
cas à un décideur (ou à un groupe de décideurs) Voir c-bas 


Le modèle prédictif répartira les cas d'Éntrée express dans trois catégories 


o Canada 3 


Marie-Josée: 


Scores and models are not everything: other rules in GCMS will be combined with the SPSS models scores so that triaging is automated 
and as accurate as possible. 


The placement of the business rules might change after we have come up with a Solution (IT) as this is just tentative for now. 
Complexity will be added in the future to the Predictive Models, in order to do complete complexity and risk-based triaging. 


Quality Assurance is a random selection of XX% of cases for QA, assigned to QA group. 

Workloads Distribution: Each bin can be assigned to officers according to operational requirements, expertise, levels, etc. 

Triage Model Validation: A random selection of cases from bin #1 and # 2 will be assigned to officers at superior levels to ensure that: 
Higher ranking officers remain exposed to all types of case. 

Senior officers can continuously validate the Triage model. 

Experienced officers can confirming that cases were effectively assigned to the right bin by the Triage model. 

Experienced field officers can detect errors in the Triage model. 

Experienced field officers can provide intelligence to centralised offices and HQ in order to improve and rafine the Triage model. 
Note: 


-or FSW, PN, CEC was used C5.0. algorithm to develop the models. Each of these LOB has different output for 
probability bins. 


oO 
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Peera des directives relotives au tage manuel iors cu lancemen 
d'Entrée express 


Surveillance continue des indicateurs de risque etde triage ala suite du 
lancement d Entree express: 


Validation continue du modèle predictiliie a Entrée express jusqua 
Tautomne 2015 


Exercice d analyse des tendances en mahere de frauds àl automne 2015 une 
. LS poxséderone suffisamment de données dress d'Entes è x 


Marie-Josée: 

Details: 

elt is anticipated to restart WG after receiving Express Entry data. 

eneed to validate model and allocation, but any feedback soon in the process will be 
appreciated. 


Ravi or David to close the deck? 


N 
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For all departments, the end goal is better allocation of resources. 


1A 
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Analytics Coalition 

Chaired by the DPAT 

Share analytics best practices with OGDs who are working on analytics projects 
Most have a similar goal: improve workload distribution. 

The 4th meeting was on May 28, 2014 
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Touch Base on Analytics Work and Transition Meeting Notes 
May 25, 2015 from 11-12 am 

Boardroom-KENT-7-95 

Attendees 


e Marie-Josée Dorion — DPAT 
e Darren Goodyear— DPAT 

e Somaieh Nikpoor— DPAT 

e Elena Tipenko— DPAT 

e Olga Anglinova - DPAT 

e Derek Pickell- PID 

e Fawad Popalyar- PID 

e Jae-Won Chung — SPO 


- Marie- Josée mentioned about SPO providing trend analysis/reports/findings (according 
to ID 258 from DPAT project’s schedule) 

- DPATto revise TR (TRV, TFW, IEC, VR-extension, Student) with addition of new derived 
variables and flags 


- Jae-Won Chung mentioned that suggested Go Live date for TR might be November, 
2016 (not January 2017 as in DPAT schedule) 

- There are no concrete deployment days for PR and Citizenship besides the years - 2017 
and 2018 correspondingly (no costing yet and uncertain number of applications for PR) 
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Analytics Coalition Meeting 
November 22, 2013 
Citizenship and Immigration Canada 


- Meeting Minutes - 


In attendance: CBSA, RCMP, ESDC, CIC 


1. CIC (Strategic 
Projects 
Office) 
overview of 
ETA initiative 


2. CIC 
Presentation & 
Discussion 


Action item: Strategic Projects Office to share Australian model of ETA. 


Scheduled launch of April, 2015 

Limited data on people that will be screened under ETA 1.0 as they 
are from non-Visa required countries. Attempting to add some level 
of risk assessment to ensure as rigorous of screening up front as 
possible. 

Will be collecting passport numbers for the first time under ETA. 
CBSA communicated that 


Question: how do you test the system for accuracy? Response: Other 
systems can assist in overall machine learning. 


Other efforts, such as dedicated resources, will also 
be devoted to quality assurance. 
CBSA: random screening also aids in machine learning. 


In preparation of ETA, looking at data for visa exempt countries from 
2008-2012 for workers and students (visa exempt countries), which 
amounts to over 500k records in total with 41 fields for each record. 
Used SPSS to select only workers, which narrowed results to 300k. 
Have to deal with outliers and missing/invalid data in 300k records. 
Conduct correlation analysis between variables in SPSS and establish 
indicators that SPSS believes are leading to the decision. 

Data has to be 
boosted or begged in order to re-weight for more accuracy. 
Autoclassified results which took average of two results. One of the 
best features is that is produces decision trees, allowing for very fluid 
clustering of data inside the model. 

l 

This should improve as business rules are improved, more recent 
data is used, model manipulation in strengthened, and from 
consulting with subject matter experts. 

Analytics for second phase of ETA (visa required countries) should be 
easier as there is already information on them as a result of the visa 
requirement. 

Second ETA will have subpopulations, such as low risk travelers from 
high risk countries. 
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e CIC: model score seems to be more frequently occurring probability 
Discussion/rountable score on the records...is this similar in other areas? 


e CIC: Do colleagues score outside or inside of production system? 


e CIC: How are indicators updated? 


e CIC: When modeling, you need business experts and rules to increase 
effectiveness of model. 
e CIC: Cache data in order to not overwhelm the system. 


e 
e 
+ 
Next Steps e Ivan to circulate distribution list of group. 
e Next meeting in January at CIC, where the 
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BACC Projects 
IEC — International Economic Class 
TR Extension 
triage 
Automated Workload Distribution 


SPO - Strategic Projects Office 


ETA 

SPO — Strategic Projects Office 

For countries where visa are not required now 
April 2015 soft launch, October 2015 hard launch 
Risk of public backlash 

Need 72 hour processing time 


Data from CBSA required 


eTraveller 


for Visa required countries 


rollout after ETA 


manage political irritants, rollout globally 


Expression of Interest (EOI) — January 2015 
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Blended approach 

Controlled intake 

Universal Points Grid 

Pool opens October 2014 

Reduce processing time 

Temporary Resident (TR) automatic decision making 
Volume increasing 

TR — worker, student 


TRP — free pass from the minister 
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Data and Predictive Analytics Team 
(DPAT), PMU/ OPMB | 


Data and Predictive 
Analytics at CIC 


Communication paper 


Prepared by the Data and Predictive Analytics Team (DPAT) in 


PMU (OPMB) 
4/30/2014 
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Data and Predictive Analytics at CIC 


There are several modernization initiatives underway at CIC to support a faster, more efficient and cost 
effective processing network. Data analytics are a key technology component underpinning these 
efforts. 


What are data and predictive analytics? 

The term data analytics refers to the practice of using a variety of statistical techniques to support 
business processes and organizational outcomes. Descriptive analytics, also known as Business 
Intelligence, provide a numeric view of the organization and operational activities via aggregate reports 
and dashboards. Analytics in this sense describes exploring data, making sense of historical events and 
discovering trends. This is the foundation for data analytics. 


Data analytics also encompasses using data to optimize and guide business decisions. Some of this work 
focuses on understanding the likelihood of an event happening in the future, which is known as 
predictive analytics. These are data driven insights that infer future events based on quantified historical 
behaviour. Predictive models are developed using statistical analysis and can be built with the capacity 
to adapt to changing conditions reflected in the data. 


Predictive analytics can take the form of a report or graphic to provide guidance to business users. Often 
the analytics are embedded into the business rules of an application, serving to optimize technology 
enabled business processes without input from business users. 


ow do predictive analytics work? 

With the aid of specialized software, a skilled data modeller builds predictive models that accepts data 
inputs and returns the likelihood of outcomes based on variables found in the data. Clarity around the 
business context produces effective results. The predictive models define a mathematically sound 
probability of outcomes. Theoretical measures, such as accuracy, precision and sensitivity are used to 
compare different models, but ultimately, the results are judged from a business perspective. While the 
future will always remain uncertain, even a small degree of foresight can have a huge impact in 
optimizing business decisions. 


Data analytics in the private industry 

Data analytics have been in use in industry since the widespread adoption of relational databases in the 
1980s. The digitization of commerce and recent advances in database technology have allowed 
companies to collect more data than ever, accelerating the use of data analytics. The results have been 
dramatic for companies that have chosen to let data analytics permeate their corporate culture. A well 
known example is Google, which offers many examples of data driven decision making at all levels of the 
organization and embedding analytic algorithms in its products. Google was founded in 1998, but it is 
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one of the fastest growing companies in history, reaching a market capitalization of $400 billion in early 
2014. 


User Experience on the Web 


An example is Google Instant Search, which returns search results as the user is typing in the keywords. 
Google uses algorithms based on the user’s location, search history and other variables to return search 
results before the user has even finished typing the search query. This enhances the user experience 
with Google’s core technology, encouraging the user to return repeatedly to use the Google search 
engine and be exposed to the advertising that Google sells. 


Fraud Prevention 


An example of using data analytics for loss prevention can be found in the financial industry: if a credit 
card user makes purchases inconsistent with the historical use of the card, then the bank is alerted to 
these possibly fraudulent transactions. The card owner is then contacted to ascertain the validity of the 
transactions. This prevents loss for both the bank and the consumer. 


ata analytics in government 
Following the examples found in the private industry, governments are leveraging the same data 


analytics technology to reduce costs, mitigate risks and promote operation effectiveness. There are 
several examples relevant to CIC: 
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Meet the DPAT: Data & Predictive Analytics Team 
The DPAT is a new team in OPMB, focusing on areas where predictive analytics can have a significant 
and timely impact at CIC. 

Dptimized case workload distribution 
will maximize the use of application processing skills at different levels, thus increasing processing 
capacity. 


Two projects are underway, one with Express Entry (EE) and the other with International Experience 
Canada (IEC). This work is a key component of the Integrated Workload Distribution (IWD) initiative for 
the Temporary Resident (TR) line of business. Predictive analytics will reduce the labour intensity of 
application triage and processing as well as minimize the time to a decision for a significant number of 
cases. A third project, still in its infancy, will help the Centralizes Processing Region (CPR) office 
consistently triage FC4 and CAN+ applications. 


ctive analytics is not 

Forecasting will take a benchmark measure and then add estimates for the future period to calculate an 
expected figure. This is useful for calculating the required capacity to deal with application demand. 
However, it is different from predictive analytics because a predictive model will test variables across 
many dimensions to calculate the likelihood of a scenario based on the relationships between the 
variables. In contrast, a forecast extrapolates a historical trend to predict future demand or output. 


OLAP cubes are a Business Intelligence (BI) technology that is often described as “analytics” or as an 
“analytical reporting” solution. The cubes are different than reports because they allow business users 
to make rapid ad-hoc queries across many dimensions (slice and dice data) and to move from summary 
to detail level data with ease. The cubes can do regression analysis and forecasting, but for most cases, 
the algorithms are too simple to produce useful results. 


Data mining is the search for hidden patterns and relationships in data sets. It helps analysts recognize 
significant facts, exceptions and anomalies that might otherwise go unnoticed. Some of the data 
modeling work necessary for predictive analytics uses data mining techniques, but data mining as a 
practice does not confine itself to results that are predictive in nature. 


Analytics are not a shortcut for validating data or casting doubt on reports from other sources. 
Statistically sound techniques exist for verifying data, but this is very different from predictive modeling. 
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ata Analytics Glossary 

Accuracy — For a predictive model, accuracy is the ratio of correct predictions to the number of cases evaluated. This is a useful 
way of comparing models during development, but it is only one criterion among many in selecting the best model to solve a 
business problem. 

Advanced Analytics — A synonym for Business Analytics, often used in reference to new technology. 

Big Data — A volume of data so great and unstructured that it cannot be stored, manipulated or analyzed using traditional 
relational database technology. Because of this volume, the analytics can lead to more interesting results than with smaller and 
less comprehensive sets of data. 


Business Analytics — Data analytics that support specific business decisions or organizational outcomes. 


Business Intelligence (BI) — Also known as descriptive analytics, BI describes the reports and infrastructure required to present 
data in a format understandable by business users . 


Data Analytics — Employing a variety of statistical techniques and software tools to explore and to extract value from raw data. 
Data Mining — Algorithmic discovery of previously unknown characteristics or patterns in a data set. 


Data Warehouse — A repository for data from operational data stores and other diverse sources. In a data warehouse, the data 
is transformed by applying business rules and the database is optimized for query and reporting purposes. 


Forecasting — Predicting a future outcome based on the extrapolation of a historical trend. 


Machine Learning — Often used as a synonym for Data Mining, but refers specifically to algorithms that evolve themselves 
based on conditions observed in the data. 


Predictive Analytics — Data driven insights that infer future events based on quantified historical behaviour. 


R — An open source programming language for statistical analysis and data visualization. Popular in academia and used by the 
DPAT. 


Relational database — A common database technology, based on the concept of data stored in tables made up of rows and 
columns, with the tables related to one another. 


SPSS — IBM Software used for statistical analysis at CIC. SPSS integrates well with IBM Cognos. 
SQL — A programming language for relational databases. 
Structured data — Numeric or text values that fit into columns and rows; metadata exists to describe the data. 


Unstructured data — Videos, photos, sounds, text or anything that has been captured and stored digitally; metadata does not 
exist and would be onerous to define. 
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CSSP Funding for Predictive Analytics 


Operations Planning and Performance Branch 
Business Research, intelligence and Analytics Directorate 
February 2017 
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Key Points: Predictive analytics project funded by CSSP 


OPPB has won a grant from the Canadian Safety and Security Program {CSSP) to undertake a 
collaborative innovation project entitled “Chain of Trust”. 


The project brings together federal, industry and academic partners to build a prototype of a 
risk assessment and passenger screening system for air travellers coming to Canada. 


OPPB will receive $237,000 in 2017-18 to use predictive analytics to develop risk indicators 
for the Electronic Travel Authorization Program {eTA}. 


The project builds an a previous CSSP-funded project titled FASTER-PrivBio. 


IRCC is the lead agency on the project. 


| What is the Canadian Safety and Security Program (CSSP)? 


The CSSP is a federally-funded competitive granting program administered by Defence Research and 
Development Canada (ORDC). The program funds government-led R&D projects in collaboration 

| with response and emergency management organizations, non-governmental agencies, industry, 

‘ and academia. 


Risk assessment will apply to all air travellers coming to Canada, which includes returning 
Canadians. 


We are the lead agency on this project. Other partners include the CBSA, three companies 
and the University of Ottawa 
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Background: The FASTER-PrivBio project 


in 2015-16, OPPB was awarded $60,000 by the CSSP for a project called FASTER-PrivBio, 
which involves the CBSA, a private company and two universities. 


The project successfully developed a prototype smartphone app that provides an 
innovative end-to-end eTA application and border management screening process. 


it leverages the capabilities of the ePassport, smartphone technology, biometrics and self- 
service kiosks. 


it also uses innovative encryption algorithms and Privacy by Design Principles to ensure 
the security of corporate and personal infermation 


This project is slated for completion in March 2017. 
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The Chain Of Trust project 


Building on this success, the Chain of Trust project aims to develop a prototype which will: 


1. Collect additional data early in the travel continuum, including: 


* Enable online submission of the E311 Declaration Form prior to arrival in Canada 


2, Consolidating all data collected about the traveller throughout the travel continuum. 


3. Develop a collaborative predictive analytics risk assessment by IRCC and the CBSA. 
* Each agency will each develop their own predictive models to assess the risk posed by travellers 


IRCE’ risk assessment during eTA adjudication will be shared with the CBSA and incorporated 
into their own risk assessment 


CBSA will update its risk assessment as each traveller advances through the travel continuum. 


OPPB’s work will focus on objective 3 


Additional info upon check-in. 
Additional info when Declaration Form filled. 
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Benefits of the Chain of Trust project 


+ for the traveller: 


» Faster processing and improved client experience for low-risk travellers 


* For program integrity: 


>» increased likelihood that high-risk individuals will be flagged for scrutiny 


* For innovation at IRCC: 


+> The project provides much-needed funds to advance predictive analytics 
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Next Steps 


Work will occur during 2017-18. 
The project charter is finalized and currently in the approval process. 


Procurement activities with private-sector partners will begin shortly, with the goal of having 
contracts in piace by April 1, 2017. 


» OPPB will receive $350,000 fram the CSSP to manage the contract with WorldReach Software 
Corporation, responsible for the improvements to the smartphone app. 


+ The CBSA will manage contracts with the other partners. 
OPPB will continue to work closely with the IPG and IRG during the development of risk indicators. 


A demonstration of the prototype is planned for early 2018. 
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Key Points: Predictive analytics project funded by CSSP 


e OPPB has won a grant from the Canadian Safety and Security Program (CSSP) to undertake a 
collaborative innovation project entitled “Chain of Trust”. 


e The project brings together federal, industry and academic partners to build a prototype of a 
risk assessment and passenger screening system for air travellers coming to Canada. 


e OPPB will receive $237,000 in 2017-18 to use predictive analytics to develop risk indicators 
for the Electronic Travel Authorization Program (eTA). 


e The project builds on a previous CSSP-funded project titled FASTER-PrivBio. 


e IRCC is the lead agency on the project. 


What is the Canadian Safety and Security Program (CSSP)? 


The CSSP is a federally-funded competitive granting program administered by Defence Research and 
Development Canada (DRDC). The program funds government-led R&D projects in collaboration 
with response and emergency management organizations, non-governmental agencies, industry, 
and academia. 


A-2018-09320-000185 


Immigration, Refugees Immigration, Réfugiés 
and Citizenship Canada et Citoyenneté Canada 
s.1 6(1 )(b) Information disclosed under the Access to Information Act 
L'information 


s.16(2) 


‘mation divulguée en vertu de la loi sur l'accès à l'information 


_ Background: The FASTER-PrivBio project 


e [In 2015-16, OPPB was awarded $60,000 by the CSSP for a project called FASTER-PrivBio, 
which involves the CBSA, a private company and two universities. 


e The project successfully developed a prototype smartphone app that provides an 
innovative end-to-end eTA application and border management screening process. 


e It leverages the capabilities of the ePassport, smartphone technology, biometrics and self- 
service kiosks. 


e It also uses innovative encryption algorithms and Privacy by Design Principles to ensure 
the security of corporate and personal information 


e This project is slated for completion in March 2017. 
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The Chain Of Trust project 
Building on this success, the Chain of Trust project aims to develop a prototype which will: 


1. Collect additional data early in the travel continuum, including: 


e Enable online submission of the E311 Declaration Form prior to arrival in Canada 


2. Consolidating all data collected about the traveller throughout the travel continuum. 


3. Develop a collaborative predictive analytics risk assessment by IRCC and the CBSA. 
e Each agency will each develop their own predictive models to assess the risk posed by travellers 


¢ IRCC’s risk assessment during eTA adjudication will be shared with the CBSA and incorporated 
into their own risk assessment 


e CBSA will update its risk assessment as each traveller advances through the travel continuum. 


OPPB’s work will focus on objective 3 
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Benefits of the Chain of Trust project 


¢ For the traveller: 


> Faster processing and improved client experience for low-risk travellers 


e For program integrity: 


> Increased likelihood that high-risk individuals will be flagged for scrutiny 


¢ For innovation at IRCC: 


> The project provides much-needed funds to advance predictive analytics 
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Next Steps 


Work will occur during 2017-18. 


The project charter is finalized and currently in the approval process. 


e Procurement activities with private-sector partners will begin shortly, with the goal of having 
contracts in place by April 1, 2017. 


> OPPB will receive $350,000 from the CSSP to manage the contract with WorldReach Software 
Corporation, responsible for the improvements to the smartphone app. 


> The CBSA will manage contracts with the other partners. 


OPPB will continue to work closely with the IPG and IRG during the development of risk indicators. 


e Ademonstration of the prototype is planned for early 2018. 
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Predictive Analytics, Business Rules, and 


BOC — For Information Date — September 14, 2015 
Presented by: Integrated Network Project, OMC 
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Introduction 


e This presentation explains — at a high level — how predictive 


analytics work and how CIC will leverage it 
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Predictive analytics — what is it? 


EEE EEE EEE EEE EEE EEE EEE EEE EE EEE 


e One Definition: “Predictive analytics is the process of Q: What is Predictive Analytics? 
discovering patterns in historical data and behaviours 
A: “... The power to predict who 


to infer the likelihood of future events and to predict Will Click buy lie. ordie: 


probable outcomes” Predictive analytics is the 
technology that learns from data 
e More simply, predictive analytics looks at how to make predictions about what 


each individual will do—from 
thriving and donating to stealing 
the information to build a model to predict future and crashing your car. By doing so, 
behaviour organizations boost the success of 
marketing, auditing, law- 
enforcing, medically treating, 


pe : | educating, and even running a 
systems. Familiar examples include forecasting the political campaign for president” 


systems - or people - behaved in the past and uses 


e Predictive analytics is widely used to model complex 


weather, predicting the performance of sports teams, 
Eric Siegel, author, former 


Columbia University professor, 
viewing history. and founder of Predictive 
Analytics World 


EE aa aa aa a  . 


and suggesting interesting Netflix titles based on your 


3 
A-2018-09320-000192 


Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


oo l 7 
| e Answers the Question | TE e Answers the Question 
“What's happening right | | | he “What might happen in 
nome 5 | i. the future?” | 
e Dashboards and other . J s # Business insights and | 
real-time monitoring 4 © predictive models | 
we 


e Answers the Question 
“What happened?” 

e Standard and ad hoc 

reports 


e Answers the Question 
“Why did it happen?” 
e After action reports, 


summaries, etc. 
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Predictive analytics — how does it work? 


Effective predictive analytics is a blend of data 
analysis, statistical modelling, and business 
intelligence 


Data scientists use powerful computer-based tools 
and statistical methods to sift through massive 
amounts of data to discover relationships, some of 
which will be obvious to business experts and others 
which may be completely new 


Business experts provide the background knowledge 
and experience to ground the model in reality by 
identifying areas where the data may be flawed, or 
business realities that might “tilt” the model in ways 
that data alone might not explain 


Data experts and business experts must work 
together to develop useful predictive models 
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_Q: Does Predictive Analytics deal 

f . 

_ causation? 


| A: "Correlation. Causation is an 
elusive thing to establish, and you 

| don't necessarily need it in order to 

| predict well. If we see the correlation 
| that early retirees have higher health 
risks, we'd like to know why — 

: but we don't actually need to know 

| why in order to make use of that 

| information. Instead, early 

| retirement becomes one factor to 

: consider when determining whether 
| to prioritize a patient for additional 

| screening or other prevention- 
oriented activities.” 


| Eric Siegel 
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Predictive analytics — a success story 


e Weather forecasting is an example of gains made as a result of predictive 
analytics 
e Weather represents a perfect arena for analytics 


— Data on weather is fairly easy to obtain 
— Data on outcomes is also easy to obtain; if we predict that it will rain tomorrow, and it doesn't, 
then we will obviously know that our prediction failed 


e Weather prediction models were once quite inaccurate, even for next-day 
forecasts 


e Weare now able to predict local weather more than a week in advance with 
high levels of accuracy 
e What changed? 


— Advances in technology, like satellites and doppler radar, have given us access to far more high 
quality data 

— Advances in computing allow us to run more complex models in reasonable times 

— Advances in our understanding have allowed us to build more accurate models 


e What was once an inexact process has become routine and dependable 
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Predictive models take historical data and 
extrapolate from it 
— Models cannot predict random developments such 
as natural disasters or regime changes 
— Models work with the data they are given 
e Ifthe data is wrong, the model will be broken 
e Ifthe data biased, the model will be biased 
e If key data is missing, the model will be flawed 
For example, if we currently streamline processing 
for a group of applicants then the data will “teach” 
the model to do the same 
— If we shouldn't be streamlining processing for that 


group (e.g. if there is an undetected program 
integrity issue) the model will perpetuate our error 


Humans can be prone to these failings as well 
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Predictive analytics — potential failure points 


| Question: Will there ever be a black President 


of the United States? 


Data for model (2008): There have been 43 US 
Presidents, and they have all been white 


Model prediction in 2008: It is highly unlikely 
that there will ever be a black President 


When Barack Obama is elected in 2009 that 
prediction became demonstrably false 


A more useful model would have included 
demographic, employment, and integration 
data to show American societal evolution — 
but the data would need to be available and 
the data experts would have had to provide it 


to the model 
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Predictive analytics — mitigating the risks 


° Models cannot be trusted unconditionally 


— As conditions evolve or unexpected events arise, even the best models may drift out of 
alignment 


— Applicants and consultants may learn to “feed the model” by showing it what it wants to see, 
leading to program integrity risks (a risk for human decision makers, as well) 


— Some applicants have unique situations which a model cannot account for 
e Risk Mitigation: 
— Before implementation, models must be tested against real applications to ensure they are 
working as designed 
— Post-implementation, models must continue to be tested to ensure that they remain valid 


— New data sources, especially data on outcomes (e.g. entry/exit information) must be actively 
sought out and integrated into our models 


— New predictive models can also be built to mitigate risks once we have better data — for 
instance, once we have entry/exit data we could design models which would predict 
whether or not particular applicants were likely to over stay 
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Predictive analytics modelling at CIC 


At present, OPMB’s Data and Predictive Analytics Team (DPAT) is working with business experts to build predictive 
models for select lines of business (eTA, Students, TRV’s, Express Entry, etc.) 


° No model is perfect 


— Inthe private sector, a predictive model which delivers a correct result more than 80% of the time is considered to be 
very effective 
CIC’s models are performing at better than industry standards in testing against historical data, but have yet to be 


tested in the field 
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Business rules — what are they? 


Predictive analytics deals in 
probabilities and likelihoods, but 
business rules are absolutes 


e A business rule is usually 


represented by logical 
statements 


it is raining you are outside 
open your umbrella 


Business rules can be very 
simple, or they can address a 
number of variables 
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| Some examples of common business rules: 


| e Different lines of business are processed in 
| select offices (IF line of business = “x” THEN 
send the application to office “y”) 


| e Applicants are also differentiated by their 
_ characteristics, like their country of 


citizenship (IF country of citizenship = “x 
THEN send the application to office “y”) 


: e Triage criteria are used to identify complex 

| clientsin many lines of business (IF the 
applicant has known admissibility concerns 
THEN flag the issue to an officer for review) 


e Triage criteria may also be used to 

| streamline files in some programs (IF the 
applicant fits the criteria THEN send their 
application to the CAN+ stream) 


Fes 
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Questions to be answered 
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Annex 1: How DPAT develops a model 


Collect Data and Business Requirements 
Conduct preliminary Data Analysis 
Develop Predictive Models 
Validate Predictive Models 


Conduct Statistical modeling of risk/complexity scoring 


Implement Predictive Models 
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Annex 1: How DPAT develops a model (continued) 


DPATuses a variety of statistical 

models such as: C5, logistical 

regression, decision trees, and 
random forest 


Predictive models perform risk 
assessment and develop risk scores 
foreach application for multiple LOBs 
(including Express Entry, eTA, IEC, and 
Students}. 


Business rules can be applied at 


Trends, classifications, and 
patterns are identified within 
the historical data 


| Predictive models 
- are tested and 

_ evaluated before 
degaynent 


In future the 


DPAT consults with ues 
Program Integrity OUTCOMES: : pe predictive models 
Division (PID) and Enable CiC’s modernization agenda Evaluation na j a 
business experts Help to detect and reduce fraud and a Hot d 
sos mupe IMSS contribute to program integrity uravé 
of business (LOBs) to pee p 
| : oe Help CIC to cope with increased volume eh dimsaives 
ne objectives aie) i 
of applications Future {Machine 


| Opportunities Learning) 


DPAT - OPMB 
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BRIA 


Predictive Analytics and Forecasting Unit 
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Activities 


Skills 


0D 0 0 0 ODO 0 


Sr 


Develop predictive models for 
workload distribution 

Use data mining to identify risk 
indicators for Program Integrity 
Run simulations to test business 
rules proposed by LoB 

Experiment with new data sources 
(e.g., socio-economic indicators) 
Work with SIMB to deploy PA 
models in real time 


Expertise in predictive modelling 
techniques 

Data interpretation and 
manipulation 

Expert skills in SPSS, R, Python and 
Excel 

Problem solving 

Understanding of operations 
Understanding of operating 
procedures 

Attention to details 

Link to business strategies 
Ability to learn new technologies 


Q 


Q) 


LRLS 


Develop forecasting models for all 
LoB except PR 

Generate/revise forecasts according 
to frequency and level of detail 
requested by LoB 

Assess sensitivity of forecasts to 
each driver (e.g. exchange rate) 

Run complex ‘what if scenarios 


Expertise in time-series 
forecasting techniques 

Data interpretation and 
manipulation 

Expert skills in SAS and Excel 
Problem solving 
Understanding of operations 
Attention to details 

Ability to learn new 
technologies 


m) 


m) 


CRARO 


Consult LoB to assess their 
needs and offer services 

Sollicit input from LoB experts 
on data issues and model results 
Provide feedback to OPMB 
colleagues on data quality and 
governance issues 

Facilitate interdepartmental 
sharing, notably through 
Analytics Coalition 

Maintain dialogue with SIMB on 
IT needs. 

Hold workshops to sensitize 
IRCC staff about PA 


Client relations and networking 
Presentation and communication skills 
Understanding of operations 
Understanding of operating procedures 
Link to business strategies 

Project management 

Data interpretation 
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PAF Outcomes - Support for Decision Making 


Assessed accuracy of various business rules proposed by Program Integrity 

Developed framework for workload optimization using Simplex algorithm 

Enhanced Passport forecasting models by incorporating new drivers (e.g. exchange rate) 

Produced monthly and annual forecasts for Passport, by LoB, office, and nationally 

Developed new models and produced annual forecasts for Temporary Resident visas, by region of origin: 
— Student visas and extensions 
— Visitor visas and extensions 
— Work permits and extensions 

Applying data mining to discover risk indicators of passport identity fraud 

Plans to develop forecasting models for additional LoB, starting with Citizenship and PR card 
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PAF Outcomes — Outreach 


° Held a dozen Working Group meetings with LoB experts to: 
- Present data trends 
—  Sollicit input on model development; and 
- Present initial model results. 


° Working with SIMB to define business requirements for real-time deployment of models 

° Developing a change management strategy for PA based on attitudes uncovered during five focus groups involving more than 
30 IRCC employees. 

° Held bilateral meetings with various branches of IRCC to raise awareness of PA 

° Hosted several meetings of the Analytics Coalition to foster knowledge sharing among analysts from a dozen federal 
departments 

° Led cost-benefit analysis of PA to support business case for real-time deployment 

° Developing dashboard for TRV model to showcase the model's capabilities and potential efficiency gains for IRCC 

° Upcoming “PA workshops” to sensitize IRCC employees to PA 
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Predictive analytics have proven value in private industry and government 


Predictive Analytics are a key piece of the modernization strategy 
DPAT is working closely with partners: SPO, PID and OMC 


CIC is at the very beginning of using predictive analytics 
We are catching up on other countries (USA and Australia for example) and other 
departments | 


Expectations need to be set carefully 
Predictive analytics can be transformative for an organization, 
but it takes a long time to put everything in place 
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Predictive Analytics ~ Overview 


What ig predictive analytics and why is it important? 

Predictive analytics at CHC 

Steps for prodictive analytics modeling 

Future Trans broush predictive analytics 
and triaging 

OPAT = Neat steps 

Date and radiau Aratya Road Man 

Annexes 1: Learn from other departments 


Annee 2: Lestaing from foreign governments 
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What is Predictive Analytics and why 


» Predictive analytics is the process of discovering patterns in historical 
data and behaviours to infer the likelihood of future events and to 
predict probable outcomes 


» Predictive analytics offers the opportunity to use data to drive decisions 


Drawing an s combination of... Benefits 


Computer Sciences » Formally substantiates business decisions made By intuition or gut 


fealing 
Statist cs 

Cambine aperational husiness knowledge and expertize with 
Gperational exotics statistical analysis 


Specialized Software Heips reduce waste by maximizing the use of resources 


Drives smarter decision making by extracting actionable insights 


from the vast quantity of data availabiein government agencies. 


Je] ee. ee Canada 


Explains definition and how it works. Also talk about concrete benefits in any business 
doing predictive analytics (Walmart, for example, or Google, that both use that now in their 
business to save money, profile their consumers and improve resource allocations) 


The benefits of combining operations expertise with empirical data and statistical analysis 
to build a predictive model: 


+ Increasing processing capacity 


+ Facilitate program integrity exercises 
* Deliver optimal client service 
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> Predictive analytics leverage statistical analysis and predictive modeling 
» Support key initiatives such as 


Express Entry and more... 


# Enable and support the modernization agenda 
s+ Contributes to CIC's toolkit far detecting fraud 


Standerd reporte 
Adde reparte 


A combination of 

manual pote 
and modernization 
initiatives 


OUTCOMES: 


Date analysis 


# Mredictive modeling 


Date is cage, iut Brera Hadicfong 


not fully leveraged ; 


FOr DUS 
purposes 


d, managed 
and satiated 


integrated Natwork Project, eTA, 


Business sudden to 
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auto grouping and faster processing. 
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Predictive Analytics at CIC (cont d) 


> The Data and Predictive Analytics Team 

» Anew team in OPMB/ PMU 
Established and funded by MCs to deliver specific projects with ongoing funds for 
maintenance activities 

established relationshios with Business intelligence (Bi) resources In 

OPMB. 
Working cl with internal partners such as OMC/ Program integrity Division (PID) 
and the Strategic Pre i 
Building partnerships wit! s and foreign governments. 


> Passgpart: 


analysis 


&® Develop the requis 
p ACJ 


# Build, validate and deploy the gredictive models 


Expertiag 


Team is new. 

When did we (CIC) get into predictive Analytics: 

TR MC and Express Entry, this year (2014) were the first projects that got approval to do 
predictive Analytics. 

TRMC: 500K app in FTE for 

Express Entry : 450K in O&M and 3 FTEs 

ETA : 2 FTEs 


Insist on Predictive Analytics capacity building will not happen overnight. Predictive 
Analytics take time to implement. 

OTHER COUNTRIES and Other Departments have done this for a few years now: proven 
value to achieve better allocation of resources. 


Building partnerships is key; 


Passport is certainly a key one we need to work with. 

And We say CIC here but a few words on passport: 

-Passport has been doing long term forecasting for years in order to forecast volume of 
work by office; and that is related to the funding model of passport, i-e revolving funds. 
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Collect Data and Business Requirements 


Conduct prediminary Data Analysis 


Develop Predictive Models 


Conduct Statistical modeling of risk/icomplexity 
scoring 


-Partners and clients: 
-OPS partners, such as OPMB-PMU, OMC-PRPD, OMC-PID, IR, CPR, and SPO. 


-Historical data: 


FSW: Total number of valid records consist of 155,665 applications 
Total number of fields is 122 unique fields 
Dates covered (final decision dates) are from 2007-01-01 to 2008-02-27 


P Ri Total number of valid records consist of 111,060 applications 
Total number of fields is 122 unique fields 
Dates covered (final decision dates) are from 2007-01-02 to 2014-08-18 


CEC: Total number of valid records consist of 25,058 applications (GCMS only) 
Total number of fields is 122 unique fields 
Dates covered (final decisian dates) are from 2011-07-16 to 2014-08-15 


Concrete benefits in any business doing predictive analytics: For example, Walmart and Googles, both use predictive analytics in their business to save 
money, profile their consumers and improve resource allocations. 


The benefits of combining operations € ise with empi 


SL EARR EULER Codd hy 


Grou 


ps 
ti 


n 


soration with partners and the business 


iD. We co-chair. 


Data Analytics Coalition 

Great forum for sharing ideas and best practices as predictive analytics is relatively new to the federal government 
Similar goals — workload distribution 

Common challenges — change management, setting users expectations, poor data quality 


Cc 
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Future Triaging and 


Business Rules 
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Predicting Final Decisions and Triaging 


Predictive Analytics have two important goals: 
1. 


For example the distribution of volumes for IEC {TR} 
(163,109 EC applications) is: 
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DPAT - Next Steps 


Work on the Stakeholder Engagement Plan for Predictive Analytics at CIC 


w byeclain predictive analytics te partrers anc 


Deliver on projects 
& 


# 


> fata Analytics Coal 


Répôrt on results to senior management 
LA 


Challenges 
Finding resources to staff the team 


Several projects running parallel, so there are limited opportunities to learn from the first projects 
and apply the lessons to other projects 


Data Analytics: reach cut 


Data Analytics Working Groups 
predictive analytics demands tight collaboration with partners and the business 
We started the WG in collaboration with PID. We co-chair. 


Data Analytics Coalition 

Great forum for sharing ideas and best practices as predictive analytics is relatively new to the 
federal government 

Similar goals — workload distribution 

Common challenges — change management, setting users expectations, poor data quality 


Conclusion 
Early days for predictive analytics at CIC 
We are catching up quickly 


Partners, such as PID and SPO, and business clients have been enthusiastic 
Important to set expectations correctly 


Questions? 


Annex slides — “Keys Steps for Success” and “Potential Pitfalls” 
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For all departments, the end goal is better allocation of resources. 


Passport 
Developed models to forecast demand. Data driven insights are used for business planning and budgeting 


OPC - Office of the Privacy Commissioner: 

Published a paper in August 2012 

http://www. priv.ge.ca/information/research-recherche/2012/pa_201208_e.asp 

Acknowledges the accelerating use of predictive analytics: 

Targeted advertising, social science, law enforcement, location tracking, fraud prevention 

Ethical guidelines are needed, with boundaries around what can be inferred from algorithms and what data can be collected for this purpose. 


Analytics Coalition 

Chaired by the DPAT 

Share analytics best practices with OGDs who are working on analytics projects 
Most have a similar goal: improve workload distribution. 

The 4th meeting was on May 28, 2014 
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Predictive analytics have proven value in private industry and government 


Predictive Analytics are a key piece of the modernization strategy 
DPAT is working closely with partners: SPO, PID and OMC 


CIC is at the very beginning of using predictive analytics 
We are catching up on other countries (USA and Australia for example) and other 
departments 


Expectations need to be set carefully 
Predictive analytics can be transformative for an organization, 
but it takes a long time to put everything in place 
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Predictive Analytics ~ Overview 


eTA Data Collection 


Th- Proxy Data Used for modelling 


Process Map 


Next Ster 
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ETA- Data Collection 


eTA-Phase 1 of Predictive Analytics = There is no real eTA data now. We are 
using the proxy data to build models to help identify some business rules that 
might be useful for the processing and triaging of the very first eTA applications. 


In between of Phase 1 and Phase 2 = Voluntarily eTA data collection. Since the 
first few months will be voluntary: 
«We think we will not have a data set that will be fully representative of 
the “real” ETA population; 
*The volume as well might not be enough to build any predictive analytics 
rules yet. 


eTA-Phase 2 of Predictive Analytics = Period of real eTA data collection. After 
some time of eTA data collection and processing of TBO% of the applications, we 
are going to have enough data (both in volume and in representativeness) to 
start building a real predictive model to deployed in production. 


Isl See. Se. Canada 
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5 LE 
We are using the proxy data to build models to help identify some business rules that might 
be useful for the processing and triaging of the very first eTA applications. 
The eTA Working Group decided, we will use the data set {si that have similar population and 
data characteristics to eTA. 


* Total number of valid records consist of 559,164 applications 
» Total number of fields is about 150 unique fields 
« Dates covered {finai decision dates) are from 2013-01-01 to 2014-11-30 


TRY (select D 
counties) 


+ Total number of valid records consist of 122,921 applications 
s Total number of fields is about 150 unique fields 
+ Dates covered (finai decision dates) are from 2011-01-01 to 2014-11-06 
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eTA- Business Rules 


Phase 1 of ETA modeling: While we don’t have the real eTA data, by using the proxy data we 
developed business rules that can be applied later to triaging the firat eTA applications. 
DPAT: 

i. Found the best model for the proxy data (TRY data} 

2. Scored the applications using this model 


3. identified business rules and discussed with the WG how we can use these rules to 
refine or fine-tune the current proposed business rules to be used at launch of eTA 


* Phase 2 of ETA modeling: After the launch of ETA we will have to wait a few months (6 to 9 
depending on if ETA starts with a voluntary period, as this will reduce the number of cases 
and will have an impact on the population most likely), we will review the models built on 
proxy data, including what was not used (citizenship and etc) and we will build real models 
te be tested and then used in production, 
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Decide on the best business rules to be used for first voluntary eTA data. 


Project eTA phase-2 will be starting in Oct 2015 (or even later, given the CR 
recently submitted to postpone ETA}, a few months after the implementation of 
the eTA (6 to 9 months depending on if ETA starts with a voluntary period) in order 
to collect real data set that will be fully representative of the “real” eTA population. 
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LE Predictive indicators and Model 


| EE Data Exploration 
Tiagra Palas for Express Entry 


rampie CEC Predictive Dashboard 


Accuracy of SPSS Triage Mules Outer vs: GEMS 


Express Entry future Teging Through Predictive Analytics 
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‘Original data set had $47,215 applications with 122 unique fleids 


Federal Skilled Worker Program (FSW) - 72.13% 
Provincial/Territarial Nominees (P/N) . 21.16% 
Canadian Experience Class (CEC) - 6.69% 
Skilled Trades (ST) - 0.03% 
Distribution by Programs 
Suled Workers 
Provincial/Teritorial Nominees T7 
Canadian Experience Class J] 
Skilled Trades] 


* Missing values and outliers were removed from the data for each LOB 

+ As a result, each LOB has reduced number of records 

+ Time periods for modeling were also chosen for each LOB depending on availability 
and quality of the data 
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* Total number of valid records consist of 155,665 applications 
* Total number of fields is 122 unique Helds 
+ Dates covered (final decision dates) are form 2007-01-01 to 2008-02-27 


* Total number of valid records consist of 111,060 applications 


«Total number of fields is 122 unique fields 
+ Dates covered (fina! decision dates) are from 2007-01-02 to 2014-08-18 


+ Total number of valid records consist of 25,058 applications (GOMIS only} 
= Total number of Helds is 122 unique elds 
s Dates taverad (ina! facing detent are freen MILAT iA tn POTTS 


For FSW, DPAT analyzed data for FSW before and after MI. As a result, before MI data was 
chosen for FSW. This decision was based on business knowledge and opinion on which 
cases should be included in the sample in the way that there would not be any lost in 
representativeness of our future population of applications, while preserving the quality of 
the data set. The fact that Express Entry does not have MI restrictions is also consistent 
with this choice of pre-MI time period for modeling. 


For PN, data was analyzed for GCMS and CAIPS separately due to some missing values 
problem. since both sets of 
data yield the same acceptance rate and adding more applications to build the model 
results in a higher degree of freedom leading to increased robustness. Thus helping to 
ensure that the model performs as expected (i.e. is not over-fitted). 


For CEC, WG suggested that DPAT use only GCMS data from 2011 to 2014-07-04 (final 
decision date) given all the missing values in CAIPS. 
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Collect Data and Business Raquiramants 


Conduct preliminary Data Analysis 


Develop Predictive Models 


Conduct Statistical modeling of risk/complexity 
scoring 


Concrete benefits in any business doing predictive analytics (Walmart, for example, or 
Googles, that both use that now in their business to save money, profile their consumers 
and improve resource allocations) 


The benefits of combining operations expertise with empirical data and statistical analysis 
to build a predictive model: 


* Increasing processing capacity 


+ raciitate program integrity exercises 
* Deliver optimal client service 


Data Analytics: out reach 


Data Analytics Working Groups 
predictive analytics demands tight collaboration with partners and the business 
We started the WG in collaboration with PID. We co-chair. 


Data Analytics Coalition 

Great forum for sharing ideas and best practices as predictive analytics is relatively new to 
the federal government 

Similar goals — workload distribution 

Common challenges — change management, setting users expectations, poor data quality 
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DPAT identified the most important 
predictors for making decision to 
approve or refuse the application. 


DPAT have looked at different 
prediction methods, such as Decision 
Trees {Le. €5.0, CHIAD...) and Logistic 
Regression, 


After developing models based on 
each of these methodologies, we 
compare the overall accuracy rate, 
discuss the results with the clients and 
select the method with the highest 
possible accuracy rate that meets CIC’s 
needs, 


It is important to focus our modeling on the fields/indicators that matter the most and 
consider dropping those that matter least. Predictor importance does not relate to model 
accuracy. 


The predictive power (non-weighted) of each recommended predictor is computed by Auto 
Data Preparation (ADP) from either a linear regression or naïve Bayes model, depending 
upon whether the target is continuous or categorical. In our case, the target FinDec is 
categorical so that Bayes naive model was used by ADP. 


The Bayes model is termed naïve because it treats all proposed prediction variables as 


being independent of one another. This is why we used naive Bayes model only for initial 
selection of predictors. 


A-2018-09320-000233 


Business rules were developed by the model for possible auto-triaging and 
auto-assignment. 


These rules can be transferred to GCMS as XML code or series of case 
statements 
an officer. 


GCMS “Answers” has being explored to triage applicants in predictive 
dashboard for CEC. 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


s.16(1)(b) 


s.16(2) 


A-2018-09320-000234 


CEC dashboard was prepared by DMT and DPAT to test business rules 
provided by the model for possible auto-triaging and auto-assignment. 
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Concrete benefits in any business doing predictive analytics (Walmart, for example, or 
Googles, that both use that now in their business to save money, profile their consumers 


and improve resource allocations) 


The benefits of combining operations expertise with empirical data and statistical analysis 
to puid a predictive model: 


kd 


* Increasing processing capacity 

& 

+ Facilitate program integrity exercises 
* Deliver optimal client service 


Data Analytics: out reach 


Data Analytics Working Groups 
predictive analytics demands tight collaboration with partners and the business 
We started the WG in collaboration with PID. We co-chair. 


Data Analytics Coalition 

Great forum for sharing ideas and best practices as predictive analytics is relatively new to 
the federal government 

Similar goals — workload distribution 

Common challenges — change management, setting users expectations, poor data quality 
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‘Original data set had $47,215 applications with 122 unique fleids 


Federal Skilled Worker Program (FSW) - 72.13% 
Provincial/Territarial Nominees (P/N) . 21.16% 
Canadian Experience Class (CEC) - 6.69% 
Skilled Trades (ST) - 0.03% 
Distribution by Programs 
Suled Workers 
Provincial/Teritorial Nominees T7 
Canadian Experience Class J] 
Skilled Trades] 


* Missing values and outliers were removed from the data for each LOB 

+ As a result, each LOB has reduced number of records 

+ Time periods for modeling were also chosen for each LOB depending on availability 
and quality of the data 
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* Total number of valid records consist of 194,665 applications 
* Total number of fields is 122 unique Helds 
+ Dates covered (fal decision dates) are fror 2007-01-01 to 2008-02-27 


* Total number of valid records consist of 111,060 applications 


«Total number of fields is 122 unique fields 
+ Dates covered {final decision dates) are from 2007-01-02 to 2014-08-18 


+ Total number of valid records consist of 25,058 applications (GOMIS only} 
= Total number of Helds is 122 unique elds 
» Dates covered (final decision dates) are from 2014-07-16 16 2014-08-15 


For FSW, DPAT analyzed data for FSW before and after MI. As a result, before MI data was 
chosen for FSW. This decision was based on business knowledge and opinion on which 
cases should be included in the sample in the way that there would not be any lost in 
representativeness of our future population of applications, while preserving the quality of 
the data set. The fact that Express Entry does not have MI restrictions is also consistent 
with this choice of pre-MI time period for modeling. 


For PN, data was analyzed for GCMS and CAIPS separately due to some missing values 
problem. since both sets of 
data yield the same acceptance rate and adding more applications to build the model 
results in a higher degree of freedom leading to increased robustness. Thus helping to 
ensure that the model performs as expected (i.e. is not over-fitted). 


For CEC, WG suggested that DPAT use only GCMS data from 2011 to 2014-07-04 (final 
decision date) given all the missing values in CAIPS. 


A-2018-09320-000242 


DPAT used Auto data Preparation methoc 
to identify the most important predictor: 
far making decision to approve or refuse 
the application 


OPAL have looked at different predictior 
methods, such as Decision Trees {i.e 
CSG, CHAD...) and Logistic Regression. 


After developing models based on each 
of these methodologies, we compare the 
overall accuracy rate, discuss the results 
with the clients and select the methoc 
with the highest possible accuracy rate 
that meets CiC’s needs. 
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It is important to focus our modeling on the fields/indicators that matter the most and 
consider dropping those that matter least. Predictor importance does not relate to model 


The predictive power (non-weighted) of each recommended predictor is computed by Auto 
Data Preparation (ADP) from either a linear regression or naïve Bayes model, depending 
upon whether the target is continuous or categorical. In our case, the target FinDec is 
categorical so that Bayes naive model was used by ADP. 


The Bayes model is termed naïve because it treats all proposed prediction variables as 
being independent of one another. This is why we used naive Bayes model only for initial 
selection of predictors. : 
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Each Express Entry application coming trough will be assessed and scored against 
predictors and as per the model that was developed and will be re-validated before 


implementation. 


The predictive mode! breaks down the Express Entry case-ioad into 3 Bins: 


Cases inthe Green Bin and Red Bin {applications scored with most certainty) will ge to in 


-Level one and Level two decision makers; 


Cases in Yellow Bin (cases with some uncertainty} will go to Level three decision makers. 


The probability bins like exam marks. Red and Green bins are passing marks for applications 
to be processed in Canada. There will be certain number of applications that pass this 
mark. 


Note: 
For FWS, PN, 


CEC was used C5.0. algorithm to develop the models. Each of these LOB has different 
output for probability bins. 
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For FSW, 85% bins design. 


For PN, 85% bins design. 


For CEC, 85% bins design. 
| 


It is expected that business and our partners will tell us us what 
the future plans are for the split in processing for CEC, CPR vs. IR. 
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Phase I: Model results are used to produce wiaping riles for design of 
Express Entry 


Wore work needs to be done in the next few months to Improve the 
poturacy and robustness of the models 


For Phase I, this is about November, 2014 


*For Phase Il, this is about 6-9 months after launching Express Entry in January, 2015 when 
Express Entry data will be available 


*Then it will be ongoing maintenance (monitoring, revision if needed) of the model 
Details: 
elt is anticipated to restart WG after receiving Express Entry data. 


*The current triage rules and risk profiles are based on the subjective input of visa officers. 
The future plan is to make these rules exhaustive and evidence-based. 
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Presentation to IR & OPS 


December 11, 2014 ( \ 0 ada 
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Predictive Analytics - Overview 


What is predictive analytics and why is it important? 


EE EE ERS sab nab no enn 


Predictive analytics at CIC 


ie ED HEH 


Express Entry Predictive Analytics Modeling 


Express Entry — Data Exploration 


Model Design 


What Can We Use Now? 


Next Steps 
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. 


Predictive analytics is the process of discovering patterns in historical 
data and behaviours to infer the likelihood of future events and to 
predict probable outcomes 


What is Predictive Analytics and wh 


ics offers the opportunity to use 


rawing on mbination of... Benefits 


>» Computer Sciences > Formally substantiates business decisions mace by intuition or gut 
feeling 
» Statistics 
>» Combine operational business knowledge and expertise with 
» Operational expertise statistical analysis 
» Specialized Software >» Helps reduce waste by maximizing the use of resources 


>» Enables the automation of business processes 


» Drives smarter decision making by extracting actionable insights 
from the vast quantity of data available in government agencies. 
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Predictive Analytics at CIC 


Predictive analytics leverage statistical analysis and predictive 


# Support key initiatives such as ntegrated workload management, 
eTA, Express Entry and more... 


» Enable and support the modernization agenda 
» Contributes to CIC’s toolkit for detecting fraud 


> Standard reports >» Data analysis » Predictivemodeling + Business rules to > Real-lume and responsive 
> Ad-hoc reports > Data is captured, but Scenario bulding ere predictive ana ytics 
ee not fully leveraged ; oer ‘pao 
» À combination of r è Risk criteria modernization goals 
for business : nes 
manual processes identified, managed ; d 


purposes 


and modernization and validated 


initiatives 


» Create opportunities for 
better client service and 
faster processing 


PredictandR 
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The Data an dictive Analytics Team 
» Anew team in OPMB/ PMU 


» Established and funded by MCs to deliver specific projects with ongoing funds for 
maintenance activities 


> Leveraging established relationships with Business intelligence (BI) resources in 
OPMB. 


» Working closely with internal partners such as OMC/ Program Integrity Division (PID) 
and the Strategic Project Office (SPO). 


Building partnerships with other departments and foreign governments. 


® Passport: 


» Passport has been doing forecasting and analytics for workload management for years. 


ictive analytics requires a long term commitment 
» Develop the requisite technical skills 

» Acquire the necessary business knowledge 

» Build, validate and deploy the predictive models 


Expertise 
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Express Entry Predictive Analytics M 


Collect Data and Business Requirements 


Conduct preliminary Data Analysis 


Develop Predictive Models 


Validate Predicative Models 


Conduct Statistical modeling of risk/complexity 
scoring 


moment reve modes 


ae 


G 


G nee et 
immigration Canada 
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Express Entry- Data Exploration 


* Total number of valid records consist of 155,665 applications 
« Total number of fields is 122 unique fields 
. Dates covered (final decision dates) are from 2007-01-01 to 2008-02-27 


* Total number of valid records consist of 111,060 applications 
* Total number of fields is 122 unique fields 
+ Dates covered (final decision dates) are from 2007-01-02 to 2014-08-18 


* Total number of valid records consist of 25,058 applications (GCMS only) 
* Total number of fields is 122 unique fields 
+ Dates covered (final decision dates) are from 2011-07-16 to 2014-08-15 


‘Citizenship and Gitoyennete at 
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Model Design 


Phase 1 vs. Phase 2: 


e Program Risk Indicators were 
tested and some adjustments 
were made to improve their 
effectiveness for Phase 1. 


e When Express Entry data 
becomes available, DPAT and 
partners will improve upon the 
Phase 1 Program Risk 
Indicators/triage criteria, 
implementing more complex 
evidence based indicators, such 
as those on this slide. 


e This should improve the accuracy 
and precision of triaging. 


‘Citizenship and Citoyenneté at 


immigration Canada Immigration Canada 
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The Future (phase 2): Automated T 


e Each Express Entry application coming trough GCMS will be assessed and scored 
against predictive model and indicators, as per the model that was developed 
with Express Entry data (draft model will be validated with 6 to 9 months of “real” 


Express Entry data). 


+ Once the case is scored, the system will auto-assign the case to a decision- 


maker (or to a group of decision-makers). 


e The predictive model will break down the Express Entry case-load into 3 Bins: 


ne decision makers 


two decision makers 


hree decision makers 
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e The plan is to use the work done on Predictive modeling so far for Phase 2, to 


feed into Phase 1 manual triaging and workload distribution. 


e The draft bins developed could be used as a “guide” for manual triaging of the 


cases. Once we have post-launch data, we can re-validate and change if need be. 


+ Some of the scenarios include the following: 


40.8% 67.2% 82.9% 
32.6% 25.4% 2.0% 
26.6% 71.5% 15.1% 


Note: See Annex 2 for details 
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Data and Predictive Analytics Team 


(DPAT), PMU/OPMB 


ic Travel 
ation (eTA) 
ng: 

Data Exploration 


Prepared by the Data and Predictive Analytics Team (DPAT) 
in PMU (OPMB) 
10/8/2014 
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1. Current situation with the data 


As it was decided by eTA Working Group, for eTA modelling work DPAT will use the data set (s) 
that have a similar population and/or the closest data characteristics. 


a. TFW and STU from visa-exempt countries (excluding US and Canada) 


b. TRVs from visa required countries (selected countries only, see Annex 5.4) 
Currently, only subset of the TFW and STU data is available for DPAT for analysis: 


e Only 2011 and 2012 GCMS data, 
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Revised Proposal - Predictive Analytics 
Deployment Strategy for ETA November 
2015 
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Original Business Proposal 


(Deployment of Predictive Analytics in the eTA program) ee 


s.16(2) 


1. A Real-time Process transfers new application data from GCMS to Predictive models 


2. Application/client related information s stored in a Data Repository. 
2a: A Batch Process transfers incremental data from GCMS to EDW-IDM every 24 hours, to integrate with 


2b: Information is fetched from other data sources 


3. The predictive model obtains identified indicators associated to the new application from the Data 
Repository. 


4. The predictive model generates three fields for each new application: 


5. This information is written back to GCMS real time for either silent mode testing or production 
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Original Business Proposal — Conceptual View 
(Deployment of Predictive Analytics in the eTA program) 


Real-time write back 
Silent mode/production 


Other Source 


Load: 24 hrs 


Other data for Predictive 


Predictive M 


ear 


Note: See next slide for details 
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New Option 
(Less intrusive workaround to solidify and enable use of Predictive Analytics 
in the eTA program in November 2015) 


1. A Batch Process transfers incremental data from GCMS to EDW every 24 hours to 
integrate with 


2.  Application/client related information is stored in a Data 
Repository. 


2a: Information related to the new data is fetched from other data sources 


3. The predictive model obtains identified indicators associated to the new application 
from the Data Repository 


4. The predictive model generates three fields for each new application: 


This information is written to EDW. 
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New Option — Conceptual View 
(Less intrusive workaround to solidify and enable use of Predictive Analytics 


> Predictive Model 
Performance Monitoring 


Note: See next slide for details 


Ag 
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Risk of not deploying a real-time predictive model in 
November 2015 — Low 


e Background 
— Wedon’t have a solid predictive model for eTA phase 1, 


e Would not deploying in real-time impact the accuracy of predictive analytics? 
— No. 


e Would it be sufficient to adopt this option without GCMS change? 


— No. The current eTA processing time is 3 minutes. This option would add at least 1 day to processing time, 
and additional processing effort from officers. 


+ What will we get out of this option? 


— Prior to significant investment in changing GCMS, we can simulate model deployment and model 
performance monitoring. 


e What do we require to implement this option? 
— Funding for acquiring SPSS Server and associated infrastructure 
— Resources to bring in additional data from other sources 
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Data requirements 


Data required for Development Phase for Predictive Models 


° So far, the following data has been provided by DMT for development of predictive model: 


— Note: For more information, refer to detailed DPAT business requirements. 


Data Required to Improve Predictive Models 


. Other information — longer term 
— Our Predictive Analytics team has not yet explored these areas, and our current 12 models used mainly CIC information so far 


— Examples of other information: 


ad 
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Predictive Analytics Modelling Current Plan 


* Go live data under revision 


Program Predictive Models Predictive Model Testing Date Tentative Go 
Developed under Development live Date * 
1 eTA eTA November 2015 | April 2016 
2 eTA eTA expansion TBD TBD 
expansion 
3 Student February 2016 June 2016 
4 TR Pilot VR Extension February 2016 June 2016 
5 IEC February 2016 June 2016 
6 Student March 2016 January 2017 
7 VR-Extension March 2016 January 2017 
8 Full TR TRV March 2016 January 2017 
9 TFW March 2016 January 2017 
10 LCP March 2016 January 2017 
11 IEC March 2016 January 2017 
12 Express Entry -CEC March 2016 June 2016 
13 Express Entry -FSW March 2016 June 2016 
|| PR 
14 Express Entry -PN March 2016 June 2016 
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Meeting with John Acheson on 


Background 


In January 2015, predictive analytics (PA) models were developed and presented for FSW, CEC and PN 
using historical data for these programs. Please see Master Document for the reference. These PA 
models were revised for FSW and CEC with addition of new historical and derogatory information for 
Express Entry actual data for 2015/03 to 2016/02 time period. For PN, there were not enough 
applications received to build PA model. It is anticipated that PN model will be developed once data 
available. 


itional Datasets 


DPAT has been provided with two new distinct data sources containing historical and derogatory 
information. 
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dels 


C5 algorithm 


PA models were built using C5 algorithm to generate predictive information as input to decision making. 
C5 builds a decision tree that works by splitting the sample based on the field that provides the 
maximum information gain. Each subsample defined by the first split is then split again, usually based on 
a different field, and the process repeats until the subsamples cannot be split any further. Finally, the 
lowest-level splits are reexamined, and those that do not contribute significantly to the value of the 
model are removed or pruned. 


Validation of PA Models 


Models are validated by partitioning the data into separate subsets or samples for the training, testing, 
and validation stages of model building. One sample is used for generating the model and others for 
testing and validation of results. 


DPAT developed an algorithm to automatically create a large number of models and chose the one with 
the highest accuracy. 


Probability Bins 


Bins accuracy is the number of cases identified by the model correctly (each bin has certain number of 
misidentified cases) given the threshold established by business experts. 
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UPDATE: FC1 and FCC Pilot in Manila and New Delhi s.16(2) 


DATA AS OF NOVEMBER 15, 2017 


OVERVIEW: 


e NHQsent 1,085 applications for the pilot in the spring of 2017; 1,046 (96.4%) files already had 
an eligibility decision by November 15, 2017. 


o Only 811 of these files were coded with the pilot “org” (results shown below on this 
page). 


o 274 were not coded with the pilot org; we do not know why. 


o Combined results for all applications in the pilot (whether they have the org or not) are 
shown on the next page. The combined results are slightly better than those in or org. 


FOR APPLICATIONS ASSIGNED TO THE “ORG” OF THE PILOT 
Manila 
e Of the 295 applications sent to Manila under the pilot and which had the “org”, 279 (94.6%) had 


an eligibilitv decision bv November 15. 2017. 
O 


New Delhi 


e Ofthe 516 applications sent to New Delhi under the pilot and which had the “org”, 509 (98.6%) 
had an eligibility decision by November 15 2017. 
o 
o 


Both offices combined 
e Overall, 788 out of 811 applications (97.2%) had an eligibility decision by November 15, 2017. 


O 
O 
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UPDATE: FC1 and FCC Pilot in Manila and New Delhi s.16(2) 


FOR ALL APPLICATIONS IN THE PILOT (WHETHER THEY HAD THE “ORG” OR NOT) 


Manila 


e Of the 399 applications sent to Manila under the pilot, 380 (95.2%) had an eligibility decision by 
November 15, 2017. 


New Delhi 


e Of the 686 applications sent to New Delhi under the pilot, 666 (97.1%) had an eligibility decision 
by November 15, 2017. 


Both offices combined 


e Overall, 1,046 out of 1,085 applications (96.4%) had an eligibility decision by November 15, 
2017. 
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UPDATE: FC1 and FCC Pilot in Manila and New Delhi ASMP) 


s.16(2) 


Data as of October 18, 2017 


OVERVIEW: 
e NHQ sent 1,085 applications for the pilot in the spring of 2017; 1,030 (94.8%) files already had 
an eligibility decision by October 18, 2017. 


o Only 808 of these files were coded with the pilot “org” (results shown below on this 
page). 


o 278 were not coded with the pilot org; we do not know why. 


o Combined results for all applications in the pilot (whether they have the org or not) are 
shown on the next page. The combined results are slightly better than those in or org. 


FOR APPLICATIONS ASSIGNED TO THE “ORG” OF THE PILOT 
Manila 
e Of the 293 applications sent to Manila under the pilot and which had the “org”, 277 (94.5%) had 


an eligibility decision by October 18, 2017. 
O 


New Delhi 


e Ofthe 515 applications sent to New Delhi under the pilot and which had the “org”, 501 (97.3%) 
had an eligibility decision by October 6, 2017. 
O 
o 


Both offices combined 
e Overall, 779 out of 808 applications (96.4%) had an eligibility decision by October 18, 2017. 


O 
O 
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UPDATE: FC1 and FCC Pilot in Manila and New Delhi 5.16(2) 


FOR ALL APPLICATIONS IN THE PILOT (WHETHER THEY HAD THE “ORG” OR NOT) 


Manila 


e Of the 397 applications sent to Manila under the pilot, 378 (95.2%) had an eligibility decision by 
October 18, 2017. 


New Delhi 


e Of the 685 applications sent to New Delhi under the pilot, 651 (95.8%) had an eligibility decision 
by October 18, 2017. 


Note: 4 of the 1,086 applications in the pilot can no longer be found in the Enterprise Data Warehouse. 


Both offices combined 


e Overall, 1,030 out of 1,086 applications (94.8%) had an eligibility decision by October 18, 2017. 
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UPDATE: FC1 and FCC Pilot in Manila and New Delhi s16(2) 


Data as of September 22, 2017 


OVERVIEW: 


e NHQ sent 1,086 applications for the pilot in the spring of 2017; 1,015 (93.5%) files already had 
an eligibility decision by September 22, 2017. 


o Only 808 of these files were coded with the pilot “org” {results shown below on this 
page). 


o 278 were not coded with the pilot org; we do not know why. 


o Combined results for all applications in the pilot (whether they have the org or not) are 
shown on the next page. The combined results are slightly better than those in or org. 


FOR APPLICATIONS ASSIGNED TO THE “ORG” OF THE PILOT 
Manila 
ə Ofthe 293 applications sent to Manila under the pilot and which had the “org”, 276 (94.2%) had 


an eligibility decision by September 22, 2017. 
O 


New Delhi 


e Ofthe 515 applications sent to New Delhi under the pilot and which had the “org”, 499 (96.9%) 
had an eligibility decision by September 22, 2017. 
O 
o 


Both offices combined 
e Overall, 775 out of 808 applications (95.9%) had an eligibility decision by September 22, 2017. 


(©) 
(©) 


A-2018-09320-000300 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


s.16(1}(b) 
UPDATE: FC1 and FCC Pilot in Manila and New Delhi s16(2) 


FOR ALL APPLICATIONS IN THE PILOT (WHETHER THEY HAD THE “ORG” OR NOT) 


Manila 


e Of the 397 applications sent to Manila under the pilot, 377 (94.9%) had an eligibility decision by 
September 22, 2017. 
O 
O 


New Delhi 
e Ofthe 685 applications sent to New Delhi under the pilot, 638 (93.1%) had an eligibility decision 
by September 22. 2017. 
O 


(©) 


Note: 4 of the 1,086 applications in the pilot can no longer be found in the Enterprise Data Warehouse. 


Both offices combined 


e Overall, 1,015 out of 1,086 applications (93.5%) had an eligibility decision by September 22, 
2017. 
O 
O 
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FC1/FCC Pilot Project for New Delhi and Manila me 
Contents 

OASIS r Nr mere ne Men ns rene Ee Ea nA ek 1 
Pilot One Patins nd ent cee dean alba a SA ee à à 3 
T: Fer both New Delhrand Manila: Een D an nn ne ro iai tes 3 
2 HOT New DIRE 2 E N Ra at A en nan At, 3 
SEATA Di E een da OE E E A E ERST 4 
Pilot May Data esineeseen nn te nets E E EE iie REEE EE E E E E EEE 6 
Lo For both New Delhi and Mama. seen ne nus carbs eaestentonnees 6 
2 POR ING W DEIB ra dim Re Re aa a sut 6 
SOT RAINE Re Ce a e M teste ne Al ri Ra aah 7 


Summary of the numbers regarding the current status of the pilot project in Manila and New 
Delhi on the FC1- Spousal /FCC - Common Law sponsorship triage criteria used by the 
Predictive Analytics Team (PAT). 


QAs: 


e What is the overall approval rate at eligibility for all applications during the same time 
period as the pilot applications were decided, based on eligibility decision date? 
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Pilot- June Data 


The results so far shows that 725 out of 738 cases already processed passed the 
eligibility requirements. In other words, mission results concurred with the triage criteria 
presented by PAT in 98.24% of the cases. 


As a result of the last run (June data), 738 cases were processed out of the 1086 


cases, representing 67.95% of the total cases sent. 339 of those cases were sent to Manila and 
687 to New Delhi. 


1. For both New Delhi and Manila: 


Distribution of Eligibility Decision for Both Missions 


Value 56 Count 
Failed |] 1.76 13 
Passe 9824 725 


Distribution of Interviews for Both Missions 


2. ror New Denn 


There were 474 cases processed out 687 cases sent to the New Delhi primary office (69%). 


Distribution of Eligibility Decision for New Delhi 
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Distribution of Interviews for New Delhi 


3. For Manila: 
There were 264 cases processed at Manila primary office out of 399 cases sent to Manila (66%). 


Distribution of Eligibility Decision for Manila 


Distribution of Interviews for Manila 
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Pilot - May Data 


For both New Delhi and Mania: 


Distribution of Eligibility Decision for Both Missions 


There were 573 cases processed by both Missions (Manila and New Delhi) out of 1086 cases 


sent by PAT. 
Out of these 584 cases, 573 passed the validation rules (98.12%). 


Value Proportion wh Count 
Passed 98.12 573 
Failed I] 1.88 11 


Distribution of Interviews for Both Missions 


There were 404 cases(“not started ”) processed by the mission at Manila primary office out of 
687 cases (“not started ”) sent by PAT. thus, until our last June run, 58.80% of the cases sent to 


New Delhi were processed. 


Distribution of Eligibility Decision for New Delhi 
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2, For Mania: 


There were 180 cases (“in progress”) processed at Manila primary office out of 295 cases (“in 
progress”) sent by PAT. It seems that 104 cases (“not started ”) sent by PAT were not processed 
by Manila office yet. In total there were 399 cases sent to Manila. 


Distribution of Eligibility Decision for Manila 
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Some notes on number of cases needed for testing or revision of 


For Express Entry, DPAT should be able to conduct the first testing and if necessary revision of the model 
by the end of June 2015 given ITA estimate of 18,300-20,300 for by the end of June 2015 and 
Admissions estimate of 7,100-7,900 for 2015. 


It was assumed that there would be approximately 4,000 admissions by the end of June (half of 
projected 7,900 admissions for 2015). Historical data from 2007-01-01 to 2014-08-15 (final decision 
date) showed that FSW were accountable for ~72% of the total for 4 programs such as FS, PN, CEC and 
ST. The 72% of 4,000 projected admissions implies 2,880 cases by the end of June, 2015. This is 
approximately 2% of the data set used in Express Entry model. The model was tested for randomly taken 
sample of 2% (2,936) of the data set and the results were proven to be reasonable and valid. 


There were five trials conducted to see if the model still results in accurate prediction for five randomly 
selected 2% samples of data set. 


A-2018-09320-000310 


Pages 311 to / à 313 
are withheld pursuant to sections 


sont retenues en vertu des articles 


16(2), 16(1)(b) 


of the Access to Information Act 


de la Loi sur l'accès à l'information 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


s.16(4)(b) 
s.16(2) 


The table showed that for 2,990 cases, SPSS predicted correctly 2,650 cases (88.63%) and 340 cases 
were predicted with less accuracy (11.37%). That is the acuracy of the model is 89%. 


The accuracy of the model varied insignificant from 86% to 89% for five randomly selected data samples. 


This indicates the robustness of the model for 2% sample of the data. 
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December 11, 2014 
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Predictive Analytics — Overvie Wh 


What is predictive analytics and why is it important? 


A i A 


Predictive analytics at CIC 


EE EEE EEE EE eee 


Express Entry Predictive Analytics Modeling 


Express Entry — Data Exploration 


The Future (phase 2): Automated Triaging 


ARAL 


Next Steps 


Annexes 1-2: Learning from other departments and foreign governments 
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Predictive analytics is the process of discovering patterns in historical 
data and behaviours to infer the likelihood of future events and to 
predict probable outcomes 


ics offers the opportunity to use d 


rawing on mbination of... Benefits 
+ Computer Sciences Formally substantiates business decisions mace by intuition or gut 
feeling 
» Statistics 
Combine operational business knowledge and expertise with 
+ Operational expertise statistical analysis 
Helps reduce waste by maximizing the use of resources 


r Specialized Software 


>» Enables the automation of business processes 


>» Drives smarter decision making by extracting actionable insights 
from the vast quantity of data available In government agencies. 
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ictive analytics leverage statistica 


# Support key initiatives such as integrated workload management, 
eTA, Express Entry and more... 

» Enable and support the modernization agenda 

» Contributes to CIC’s toolkit for detecting fraud 


> Standard reports >» Data analysis » Predictive modeling > Business rules to > Real-lume and responsive 
> Ad-hoc reports >» Deta is captured, but Scenario bulding manage wordoad predictive anaytics 
and achieve 


» À combination of nor fully leveraged è Risk criteria mocernization goals 
for business 


manual processes identified, managed d 
Tenue purposes H > 

and modernization and validated 

initiatives 


>» Greate opportunities for 
better client service and 
faster processing 


Predict and R 
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Predictive An 


» Anew team in OPMB/ P 


» Established and funded b Cst 


maintenance activities 


> Leveraging established relatio 
OPMB. 
» Working closely with internal 
and the Strategic Project Offic 


shi 


» Building partnerships with o 
® Passport: 


# Passport has been doing forecasting and analytics for workload management for ee 


» Develop the requisite technical skills 
» Acquire the necessary business knowledge 
» Build, validate and deploy the predictive models 


Predictive analytics requires a long term commitment 


à Immigration, Refugees 
and Citizenship Canada et Cito 


Information disclosed under the Access to Infor 


deliver specific projects with ongoing funds for 


edictive Analytics Team 


3 


swit siness Intelligence (BI) resources in 


ers such as OMC/ Program Integrity Division (PID) 
0). 


epartments and foreign governments. 


nalysis 


Predictive 
Analytics 


| 


Expertise 4 
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A a 


Collect Data and Business Requirements 


Conduct preliminary Data Analysis 


Develop Predictive Models 


Validate Predicative Models 


Conduct Statistical modeling of risk/complexity 
scoring 


eee 
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e Total number of valid records consist of 155,665 applications 
- Total number of fields is 122 unique fields 
+ Dates covered (final decision dates) are from 2007-01-01 to 2008-02-27 


- Total number of valid records consist of 111,060 applications 
* Total number of fields is 122 unique fields 
* Dates covered (final decision dates) are from 2007-01-02 to 2014-08-18 


- Total number of valid records consist of 25,058 applications (GCMS only) 
- Total number of fields is 122 unique fields 
+ Dates covered (final decision dates) are from 2011-07-16 to 2014-08-15 
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* Each Express Entry application coming through GCMS will have business rules applied then 
scored against predictive model and indicators, as per the model that was developed with 


Express Entry data (draft model will be validated with 6 to 9 months of “real” Express Entry 
data). 


The predictive model will break down the Express Entry case-load into 3 Bins 


Of note: the bins and the colors do not necessarily mean workload distribution. The allocation can be 
adjusted based on CIC’s decisions. 
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Annex 1: Learning from Oth 


Canadian 


Cargo inspection 


ccès à l'information 
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December 2014 


SK or Hubert: 
Predictive analytics have proven value in private industry and government 


Predictive Analytics are a key piece of the modernization strategy 
DPAT is working closely with partners: SPO, PID and OMC 


CIC is at the very beginning of using predictive analytics 
We are catching up on other countries (USA and Australia for example) and other 
departments 


Expectations need to be set carefully 
Predictive analytics can be transformative for an organization, 
but it takes a long time to put everything in place 


1 
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Predictive Analytics - Overview 


‘What is predictive analytics and wy |x it rancrtant? 


Predictive analytics at CIC 


Express Entity Predictive Anatytics Modeling 


Eapress Entry ~ Date Exploration 

Express Entry- Predictive indicators one! Model 
Predicting Final Decision and Thaging 

Express Entry: Neat St 


hanexes 1.2: kaante fours other devattments 
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Predictive analytics is the process of discavering patterns in historical 
data and behaviours to infer the likelihood of future events and to 
predict probable outcomes 


» Predictive analytics offers the opportunity to use data to drive decisions 


Drawing ona combination of... Benefits 


Computer Sciences 


MJ opens and explains definition and how it works. Also talk about concrete benefits in 
any business doing predictive analytics (Walmart, for example, or Google, that both use 
that now in their business to save money, profile their consumers and improve 
resource allocations) 


The benefits of combining operations expertise with empirical data and statistical 
analysis to build a predictive model: 


* Increasing processing capacity 


* Facilitate program integrity exercises 
* Deliver optimal client service 


How will predictive analytics help to detect fraud?: 
e First, we will identify cases where we already know there is fraud 


e Next, we will apply analytics to assess the cases for common characteristics 

e Once we identify these characteristics, we will develop business rules to asses new 
cases to see if they share these characteristics — if they do, we will flag them for 
detailed examination 
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» Predictive analytics leverage statistical analysis and predictive modeling 
®» Support key initiatives such as automated decision making, integrated workload management, 
STA, Express Entry and more... 
Enable and support the modernization agenda 
Contributes to CIC's toolkit for detecting fraud 


hata is captured, bat 
na tally leveraged 
fat business i 
parposis 


& combination of 
ÉLUS MERRE 
and ocean 


taatad mens amg 
initiaties ents 


decision making 

tules 
Cane appariunities is 
bete cleat servite as 
faner grades 


Predictand Respond 


~ n egie Cree 


MJ: 


Read the deck bullets and insist on OUTCOMES: : auto grouping 
and faster processing. 
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The Data and Predictive Analytics Team 
» A new team in OPMS PMU 


Est hé ed by MCs to deliver 


: ip 
Passport: 


» Pasaporthasbeds doing forecasting ated analytics for gui range Tory 


» Predictive analytics requires a long term cormmitment 


the regulise technical skills 


Team is new. 

When did we (CIC) get into predictive Analytics: 

TR MC and Express Entry, this year (2014) were the first projects that got approval to 
do predictive Analytics. 

TRMC: SOOK app in FTE for 

Express Entry : 450K in O&M and 3 FTEs 

ETA : 2 FTEs 


Insist on Predictive Analytics capacity building will not happen overnight. Predictive 
Analytics take time to implement. 

OTHER COUNTRIES and Other Departments have done this for a few years now: 
proven value to achieve better allocation of resources. 


Building partnerships is key; 


Passport is certainly a key one we need to work with. 

And We say CIC here but a few words on passport: 

-Passport has been doing long term forecasting for years in order to forecast volume of 
work by office; and that is related to the funding model of passport, i-e revolving funds. 
-Passport has not yet done analytics on decision making for applications. 
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Concrete benefits in any business doing predictive analytics (Walmart, for example, or 
Googles, that both use that now in their business to save money, profile their consumers and 
improve resource allocations) 


The benefits of combining operations expertise with empirical data and statistical analysis to 
build a predictive model: 


e increasing processing capacity 
+ Consistency of application decisions 


+ Deliver optimal client service 


Data Analytics: out reach 


Data Analytics Working Groups 
predictive analytics demands tight collaboration with partners and the business 
We started the WG in collaboration with PID. We co-chair. 


Data Analytics Coalition 

Great forum for sharing ideas and best practices as predictive analytics is relatively new to the 
federal government 

Similar goals — workload distribution 

Common challenges — change management, setting users expectations, poor data quality 
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Express Entry- Data Exploration 
~ Total number of valid records consist of 185,665 applications 
» Total number of fields 122 unique tekts 
+ Dates covered (inal decision dates} are from 2007-01-01 to 2008-02-27 
| + Total number of valid records consist of 111,060 appications 
. Total number of fields is 122 unique fiekds 
+ Dates covered (inal deci dates) are frome 2007-01-02 to 2014-08-18 
Total number of valid records consist of 26,058 appicatons {BOMS only} 
Yotal number of fields is 122 unique fekis 
Dates covered {final decision dates) are trom 2011-07-16 io 2014-08-15 
For FSW, DPAT analyzed data for FSW before and after MI. As a result, before MI data 
was chosen for FSW. This decision was based on business knowledge and opinion on 
which cases should be included in the sample in the way that there would not be any 
lost in representativeness of our future population of applications, while preserving the 
quality of the data set. The fact that Express Entry does not have MI restrictions is also 
consistent with this choice of pre-MI time period for modeling. 
For PN, data was analyzed for GCMS and CAIPS separately due to some missing values 
problem. since both 
sets of data yield the same acceptance rate and adding more applications to build the 
model results in a higher degree of freedom leading to increased robustness. Thus 
helping to ensure that the model performs as expected (i.e. is not over-fitted). 
For CEC, WG suggested that DPAT use only GCMS data from 2011 to 2014-07-04 (final 
decision date) given all the missing values in CAIPS. 
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OPAT used Auto data Preparation 
method to identify the most important 
predictors for making decision to 
approve or refuse the application. 


DPAT have looked at different 
prediction methods, such as Decision 
Trees (ie. C50, CHIAD...) and 


Logistic Regression. 


After developing models based on 
each of these methodologies. we 
compare the overall accuracy rate, 
discuss the results with the clients 
and select the method with the 
highest possible accuracy rate that 
meets CIC's needs. 


It is important to focus our modeling on the fields/indicators that matter the most and 
consider dropping those that matter least. Predictor importance does not relate to 
model accuracy. 


The predictive power (non-weighted) of each recommended predictor is computed by 
Auto Data Preparation (ADP) from either a linear regression or naive Bayes model, 
depending upon whether the target is continuous or categorical. In our case, the target 
FinDec is categorical so that Bayes naive model was used by ADP. 


The Bayes model is termed naïve because it treats all proposed prediction variables as 
being independent of one another. This is why we used naive Bayes model only for 
initial selection of predictors. 
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Predicting Final Decisions and Triagil | 


Each Express Entry application coming trough will be assessed and scored against 
predictors and as per the mode that was developed and will be re-validated before 


implementation. 


The predictive model breaks down the Express Entry case-ioad into 3 Bins: 


de Level one derision makers 


level two decision makers 


Cases in the Green Bin and Red Bin (applications scored with most certainty} will go to in — 


Level one and Level two decision makers; 


Cases in Yellow Bin (cases with some uncertainty} will go to Level three decision makers. 


Canada 


The probability bins like exam marks. Red and Green bins are passing marks for 
applications to be processed in Canada. There will be certain number of applications 
that pass this mark. This number of applications depends on the threshold (in our case 
85% for PN and CEC and 80% for FSW) that is established in agreement with business 
and our partners. Yellow bins means that the probability for FinDec to be approved or 
failed is not that certain (between 60% and 85% in our case) so that applications from 
these bins will go for processing overseas for FSW and PN. For CEC, 99% applications is 


processed in Canada so that cases corresponding to Yellow bin will go for the additional 
QA. 


Note: For each application that comes into GCMS, the Predictive Analytics Models will 
produce the probability that predicted FinDec will be approved or refused. For FWS, 
PN, CEC was used C5.0. algorithm to develop the models. Each of these LOB has 
different output for probability bins. 
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Note: The accuracy of the modeidoes not depend onthe choice of the probability 


threshold 


Canada 
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Express Entry - Next Steps 


Phase |: Model results are used to produce triaging rules for 
design of Express Entry 


More work needs to be done in the next few months to improve 
the accuracy and robustness of the models 


Phase H : Models will be rebuilt to produce tiaging rules for 
design of Express Entry using Express Entry data 


Predictive Analytics on Express Entry will include the future 
vision of Integrated Workload Distribution 


C a da 1 


For Phase I, this is about November, 2014 


*For Phase Il, this is about 6-9 months after launching Express Entry in January, 2015 
when Express Entry data will be available 


*Then it will be ongoing maintenance (monitoring, revision if needed) of the model 
Details: 


elt is anticipated to restart WG after receiving Express Entry data. 

elt is also planned to work with GCMS to develop Answers Dashboard for possible auto- 
triaging and auto-assignment. 

*The current triage rules and risk profiles are based on the subjective input of visa 
officers. 

*The future plan is to make these rules exhaustive and evidence-based. 
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Annex 1: Learning from Other Depar 
Canadian Border Services Agency (CBSA) 


For all departments, the end goal is better allocation of resources. 


Analytics Coalition 
Chaired by the DPAT 


Share analytics best practices with OGDs who are working on analytics projects 
Most have a similar goal: improve workload distribution. 
The 4th meeting was on May 28, 2014 
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Hubert/ SK to mention 
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Express Entry Data Analytics sub WG!. Held December 3'4, 2014 


General notes 


WG discussed prepared by DPAT master document that combined DPAT’s data exploration and 
modeling for three LoBs FSW, PN and CEC. It was decided to change the wording for the Bins renaming 
green, red and yellow bins as high probability of approval, high probability of refusal and final decision 
probability is uncertain correspondingly. 


It was mentioned that previous referral to CIC’s future plans to split processing between in-Canada and 
Overseas, namely 60-40-split should be reconsidered. 


WG didn’t have time left to go over all Decision Trees in the document. 


Randy suggested adding definition of the “mode”. 


Randy provided his suggestions for some wording in the document regarding how to address FinDec 
(passed FinDec vs. FinDec approved) and some other comments. 


Action ilenis 


DPAT changed wording for bins according to WG suggestions 


Business rules were developed as parts of decision tree for CEC for WG consideration 
Mode definition was included to the master document 


sd a a 


Randy’s comments and suggestions were reflected in the master document 
Next steps 


Develop a full set business rules that we can use for triaging rules 


De 


3. Continue to work on modeling for Express Entry LoBs 
4. Continue to work on master document 
5. Determine how many applications we need to build a model based on EE data 


1 See invite from the meeting for attendees, but these groups were represented: OPMB-DPAT, OMC-PID, OMC- 
TRPD, IR, and SPO. 
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6. Discuss and agree on final placement of Predictive Analytics Model in GCMS 
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Express Entry Data Analytics sub WG. Held January 19", 2015 


General notes 


WG closed the First Phase of work on predictive analytics for Express Entry as described in 
corresponding final master document. It was mentioned that the work gets in the “cloud state” waiting 
for actual data to come for model to be tested and if necessary revised at spring/summer time period. 
The first two weeks data after the launch of Express Entry showed that there was a chance that the 
profile of new applicant would match MI1 profile. It also showed that CEC wasn’t the smallest LoB to 


apply. 
Action items 


1. DPAT toned down the sentence regarding security screening as suggested by OMC and PRPD 
2. DPAT sent final master document to WG 
3. Send out the last EE Dashboard, and get all EE-WG members on the distribution list 


Next steps 


2. Determine how many applications we need to build a model based on EE data 
3. Possible engagement of senior management in the discussion on final placement of Predictive 
Analytics Model in GCMS 


1 See invite from the meeting for attendees, but these groups were represented: OPMB-DPAT, OMC-PID, OMC- 
TRPD, IR, and SPO. 
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Data and 
Predictive 
Analytics Team 
(DPAT), PMU/ OPMB 


Justification of 
ion for eTA 
ionnaire 


Prepared by the Data and Predictive Analytics Team (DPAT) 
in PMU (OPMB) 
9/17/2014 
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1. Purpose 


The purpose of this document is to justify the importance of inclusion of EDUCATION level attainment 
into eTA questionnaire. In this document EDUCATION was presented in the same way as it was defined 
for FSW (includes different levels of education such as None, Secondary or Less, Trade Certificate, Non- 
University Certificate, Post-Graduate — no degree, University — no degree, Bachelor degree, Master’s 
degree, Doctorate) and STUDENTS (STU) (includes different levels of education such as Secondary or 
Less, Trade, Other Post-Secondary, University, Other). 


We were also asked to investigate the importance of educational institution, start date, end date, city 
and country of education. However’, as the data is not always or not at all in GCMS fields (but rather in 
Schedule 1 info) or we do not have it at all (for students), we did not try to make up numbers or 
justifications. We could however say that information on education (dates from and to, and institution) 
could certainly add value to modeling as we have seen other similar data points offering good indicators. 


1.1. The eTA application 


In order to simplify the process and meet travellers’ expectations, the eTA application is designed to be 
very simple with smart/filtered questionnaire flow. It means that the number of indicators for the 
automated approval process might well be very limited. We also have to make sure that a streamline 
application process does not impede CIC's ability to use the data collected from eTA clients to do Data 
Analytics and Predictive Analytics (or any other automation tools that we will use in the future). 


Jhy is EDUCATION important? 


It is important to understand that there is no historical data available for eTA to identify predictors at 
this moment. However, it is possible to use some other business lines’ data as a proxy before the launch 
of eTA since they collect similar information on the applicants. 


We suggest using the following two data sets to evaluate the significance of the Education level 
indicator in making decision approved/refused for FinDec: 


1. FSW for period from 2007-01-01 to 2009-01-01 (final decision date) and 
2. Students (STU) for period from 2008-01-01 to 2012-21-31 (final decision date). 


1 Also note that this document was put together in one day, so we did not want to try to overdo it and give 
misleading information. 
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Predictor’s importance and the accuracy of prediction for FinDec were assessed with and without s.16(1)(b) 


Education levels. Note that other Education information such as study period, city and country were not 
available (at least at this point) for both FSW and STU. Institution information was available only for STU 
but was not used for this exercise. The results were compared and presented in the charts and tables 
together with descriptive statistics illustrating the significant correlation between Education and FinDec 
to confirm the importance of Education to be included in the application form. 


s.16(2) 


3,1 FSW (2007-01-01 to 2009-01-01) 


Predictor importance analysis and modelling were conducted on FSW data with and without education 


included. 


3.2 STU (2008-01-01 to 2012-21-31) 
Predictor importance analysis and modelling were conducted on STU data with and without education 
included. 
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3. Conclusion 


As it was indicated above, EDUCATION level is an important predictor for the Predictive Analytics 
modelling process for FinDec. This is consistent with PID recommendations that include Education as a 
risk indicator for FSW and STU. As we could see for two lines of business, such as FSW and STU, 


Since the number of indicators for the eTA automated approval process is very limited, removing such 
an important indicator as EDUCATION will lead to significant information lost and have an impact on 
Predictive Analytics accuracy. 
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descriptive statistics 
a. Those who claimed refugee status before 2009 and never applied after 
i. Some of them were approved 
ii. Some of them were removed from Canada 
b. Those who claimed refugee status before 2009 and made applications after 
c. Those who claimed refugee status after submitting some applications 


Statistics on those who claimed refugee status and have never applied for a 
visa 
From Refugee data 


The variables we have: 


Statistics on those who claimed refugee status and applied for a visa after 

There are 4599 clients that have had both refugee claims and applications. Of them, 3807 clients 
claimed refugee status before submitting any visa applications and only 792 claimed refugee status after 
submitting some application: 


e _181- some app before, no app after 
e 611 -some app before, some app after — take only some for the model 


354 records with refugee claims to be analyzed for the model 


A-2018-09320-000350 


Pages 351 to / à 362 
are withheld pursuant to sections 


sont retenues en vertu des articles 


16(2), 16(1)(b) 


of the Access to Information Act 


de la Loi sur l'accès à l'information 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


s.16(1)(b) 
s.16(2) 


Predictive Analytics 


Predictive analytics describes the process of discovering patterns in past behavior to predict the 
outcome of future events. It combines computer science, statistics and operations research to drive 
smarter decisions by extracting actionable insights from vast quantities of data. Predictive analytics can 
help to prevent fraud and reduce waste by enabling the automation of manual business processes and 
routine decision making. Aside from the operational efficiencies gained, predictive analytics can also 
be used to standardize decision making processes, increasing fairness and transparency. 


myu PETE 


Timelines. Milestones. and Denenclencies 


Milestones tor ditterent models vary, based on Individual project commitments. Ine 
following list highlights current work: 


e Models for Automated triaging and auto-assignment of Express Entry applications for Fall 2015 for 
Phase 2 of modeling (working with real express entry data), for a roll out of a Predictive Analytics 
model in late spring 2016 


e Business rules for triaging of eTA applications developed in February 2015 (phase 1 of the eTA 
work}. 


Operational implementation of the models is critically dependent upon integrating the team’s work with 
that of SIMB, as data exchange with GCMS is crucial to apply the models to applications in progress. 


Level of Effort 
Moderate - The Data and Predictive Analytics Team, officially formed in March 2014, currently has 6.5 
FTE's 

A significant amount of work has 
been undertaken to acquire the necessary software tools and develop internal expertise. 


Lead Group 
e Data and Predictive Analytics Unit, Operational Performance Management Branch 
Key Supporting Groups in CC 


e Solutions and Information Management Branch 

e Programs, Operational Management and Coordination 

e Program Integrity Division, Operational Management and Coordination 

e = International, Central Processing, and Domestic Regions 

e Strategic Projects Office 

e Performance Management Unit, Operational Performance Management Branch 
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Visitor screening: accuracy necessitates complexity ome 


: | | | = The Data and 
Predictive Analytics Team (DPAT) attempted to improve current eTA system to screen 
l by applying advanced statistical methods to data about TRV visitors 
between 2010 and 2015. 


DPAT investigated both of these approaches by applying advanced statistical methods 
to historical | TRV data. 


The first approach would be ideal, if it would be possible to create a small number of 
simple and accurate risk indicators to quickly flag high-risk applications. DPAT 
employed a type of model which is designed to generate a small number of simple rules 
(a “Decision List” model). We found that the rules generated by the model were simple 
and intuitive, but they would fail to identify a significant portion of high-risk applicants. 


The second approach is to employ a predictive analytics model (C.5 decision tree). © 


Note: both approaches might require additional questions included in the eTA 
questionnaire. 
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What is Predictive Analytics? 


Generally speaking, predictive analytics uses computers to analyze past behaviors 
to discover patterns that enable predictions of future behaviours. 


Predictive analytics 
draws upon a 
combination of 


k 


* Advanced Statistics 
* Computer Science 
+ Business Expertise 


+ Specialized Software 


Using these assets, 
trends within data sets 
can be uncovered and 

linked ta outcomes. 


An evidence-based 
decision making mode! 
that substantiates 
dacisions made by 
human intuition. 
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Predictive Analytics Is Being Increasingly 


Predictive analytics is being successfully used in the private, public, and non-profit 
sectors to enhance how organizations serve their clients. Some examples: 


é 
= 


ie ne 
Ne 


t 


3 


Vancouver Police Department F p 
Medway Youth Tru 
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Predictive Analytics at IRCC 
At iRCC, predictive models are being developed to: 
— Triage applications between high, medium and low risk. 
— Support fraud detection. 
A model is built by analyzing thousands of past applications, along with almost all other 
information available to officers at the time a decision is made. 
The computer analyzes how combinations of applicant characteristics correlate with 
approval, refusal, fraud, etc. 
The strongest and most convincing correlations are used to inform decision making. 
At IRCC, predictive models are being developed to: 
Rather than just identify low-risk cases, we could triage all 
applications between high, medium and low risk. 
Support fraud detection. We’re currently working on this for CIT 


an: 


A model is built by analyzing thousands of past applications, along 
with almost all other information available to officers at the time a 
decision is made. 


The computer analyzes how combinations of applicant 
characteristics correlate with approval, refusal, fraud, etc. 


The strongest and most convincing correlations are used to inform 
decision making. 
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Benefits of Predictive Analytics at IRCC 


— Focus on complex cases 
— Strengthen program integrity 


— Reduce backlogs 


Models can support program integrity directly by 
flagging high-risk cases for greater scrutiny. 


in the future, we will try to predict undesirable 
behaviour (e.g. refugee claims) 


Focus on complex cases 
Strengthen program integrity 
Reduce backlogs 


Models can also flag the highest-risk cases that merit extra scrutiny. I'll 
show you an example of what we’re doing in this area. 


In the future, we aim to predict undesirable behaviours (e.g. criminality 
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A one-day pilot was conducted on December 16, 2016. 


DM and DMA were very pleased with results, 


ADM Orr has asked for an action plan before the end of March. 
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Example 2: Risk Triage of Spousal Sponsorships 
in June 2016, predictive models were used to perform a risk triage of the entire in-Canada 
spousal inventory. 


Each application was assigned a rating of high, medium or low risk, to assist officers when 
deciding how much scrutiny may be required. 


Among finalized applications since June, those that were identified as low risk had a refusal 
rate of just 4.7%, while high-risk cases had a refusal rate of 24.6%. 


Refusal Rate by Risk Level 


Average refusal 
rote: 8.838 


Low Risk Medium Risk High Risk 


‘re working toward a pilot for Overseas spousal sponsorships 
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Over the past year, we've worked with PID on a proof of concept to determine if 
precictive analytics can help us find risk Indicators for Passport identity fraud. 


The proof of concent was successful. The model was able to identify pockets of nigh risk 
people and sockets of very low risk people. 


Here is an example of a risk indicator for high-risk applicants... 


These findings are preliminary; this is just a proof of concept, so we're not 
recommending that this indicator be used in the field, but it gives an idea of the kind of 
atterns that a el can recognize that might not jump out to the naked eye. 


Wer ings thing similar for CITZ 
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Example 4: Flagging High Risk Cases for € 


Predictive models can be used to 
strengthen program integrity by 


identifying high-risk cases. Percentage of high-risk eTA cases 


detected, by targeting approach 


For eTA of ationals, 30 

preliminary results suggest that a 26.0 
predictive model detects more than 25 

five times as many high-risk clients 

compared to quality assurance done 

through random selection. 


Better targeting can increase the 


“hit rate” and reduce the amount of 
resources dedicated to screening. 


Random checks Predictive modei 


My last example pertains to ETA. As you're probably aware, was à big day 
for ETA because became: to 
visit Canada. 


PIB has asked us to work with them to identify 
indicators of high-risk applications that should crop out of the automated process and be 
manually reviewed, 


Based on very preliminary work, if we used a model to pick high risk applications, about 
iin 4 would be from a client for whom we would refuse entry inte Canada. Thats much 
higher than if we just picked applications randomly, in which case less than 5% would be 
refused entry. 
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Business Rules and Predictive Analytics 


Business rules can be a great tool, but they are not the same as predictive analytics. 


Suppose we're trying to identify low-risk applications: 


Rusinegss Rules 


Business rules are Increasingly being used to expedite processing. 
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A well designed business rule can be a great tool. But ivs not the same as predictive 


analytics. 
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Pros and Cons of Predictive Analytics Compared to Busine 


: Advantages 


+ Exhaustive. Models consider how all available * More complex. Predictive analytics requires 
information correlates with the outcome. Business specialized skills and software. 


rules are limited by our imagination. : 
+ Higher upfront cost. Models take more time and 


Granularity, Models create rules with many resources to create and deploy. However: 
conditions that target very similar cases. 

~ {T infrastructure required for predictive models 
Flexibility. Business rules that are hard-coded into a 
system cannot be updated regularly; models can be 
updated more easily. 


* informative. Models use probability scores to 
specify each the degree of risk, rather than just a 
“pass” or “fail. 


Exhaustive: Models consider how all available information correlates 
with the outcome. it draws on the main strength of computers, W 
is their ability to do millions of calculations per second. The cor 
will try all the combinations of applicant characteristics and see ho: 
they correlate with the predicted outcome. Business rules are limited by 
our imagination. 


ranularity: 
e Models create rules with many conditions that target very similar 
cases. 
* The granularity offers potential for greater precision in targeti 
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Our Next Steps for Citizenship 


Finish developing a model identifying low- 
risk applications 


Run tests and simulations of the mode! 


Design a pilot to test the model in the real 
world 


If the pilot is successful, examine how the 
model can be integrated into the business 
process 
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For more information... 


Steven Gonzalez 
Assistant Director, Predictive Analytics Team 
Business Research, Intelligence and Analytics Directorate (OPPB) 


Steven. Gonzalez@cic.gr.ca 
613-437-5556 


At IRCC, predictive models are being developed to: 


* Rather than just identify low-risk cases, we could triage all 
applications between high, medium and low risk. 


rking on this for CIT 


e Support fraud detection. We're currently 


A model is built by analyzing thousands of past applications, along 
with almost all other information available to officers at the time a 
decision is made. 


The computer analyzes how combinations of applicant 
characteristics correlate with approval, refusal, fraud, etc. 


The strongest and most convincing correlations are used to inform 
decision making. 
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What is Predictive Analytics? 


Generally speaking, predictive analytics uses computers to analyze past behaviors 
to discover patterns that enable predictions of future behaviours. 


Predictive analytics 
draws upon a 
combination of 


k 


* Advanced Statistics 
* Computer Science 
+ Business Expertise 


+ Specialized Software 


Using these assets, 
trends within data sets 
can be uncovered and 

linked ta outcomes. 


An evidence-based 
decision making mode! 
that substantiates 
dacisions made by 
human intuition. 
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Predictive Analytics at IRCC 


At iRCC, predictive models are being developed to: 


— Triage applications between high, medium and low risk. 
— Support fraud detection. 


A model is built by analyzing thousands of past applications, along with almost all other 
information available to officers at the time a decision is made. 


The computer analyzes how combinations of applicant characteristics correlate with 
approval, refusal, fraud, etc. 


The strongest and most convincing correlations are used to inform decision making. 


At IRCC, predictive models are being developed to: 


Rather than just identify low-risk cases, we could triage all 
applications between high, medium and low risk. 


Support fraud detection. We’re currently working on this for CIT 


an: 


A model is built by analyzing thousands of past applications, along 
with almost all other information available to officers at the time a 
decision is made. 


The computer analyzes how combinations of applicant 
characteristics correlate with approval, refusal, fraud, etc. 


The strongest and most convincing correlations are used to inform 
decision making. 
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Benefits of Predictive Analytics at IRCC 


Officers can be freed up to: 
— Focus on complex cases 
~ Strengthen program integrity 


— Reduce backlogs 


Models can support program integrity directly by 
flagging high-risk cases for greater scrutiny. 


in the future, we will try to predict undesirable 
behaviour (e.g. refugee claims) 


Focus on complex cases 
Strengthen program integrity 


Reduce backlogs 


Models can also flag the highest-risk cases that merit extra scrutiny. I'll 
show you an example of what we’re doing in this area. 


In the future, we aim to predict undesirable behaviours (e.g. criminality 
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Overview of IRCC Predictive Analytics Proj 


Senior management approved S10M over 3 years to build and deploy predictive analytics models across 
several lines of business. 


1. Deploy models to support processing of visas and student permits for China and India, as a first step 
towards à global solution: 
a. Daily risk triage of applications. 


2. Predictive models to support risk-triage and fraud detection for Citizenship and Passport applications. 


3. Develop a Business Intelligence and Exploratory Environment {i.e. sandbox} to support the project. 
Other Predictive Modeling Projects 
Proof of concept to identify high-risk applicants and trends for Electronic Travel Authorizations (eTA). 
Classification models to triage and centralize processing of low-risk spousal sponsorships. 
Network analysis to link known fraudsters to other clients 


Assessing IRCC client service using sentiment analysis (text mining) 
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+ Aone-day pilot was conducted on December 16, 2016. 


+ Next pilot scheduled for October 30 to November 17, 2017 
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Example 2: Risk Triage of Spousal Sponsorships 
in June 2016, predictive models were used to perform a risk triage of the entire in-Canada 
spousal inventory. 


Each application was assigned a rating of high, medium or low risk, to assist officers when 
deciding how much scrutiny may be required. 


Among finalized applications since June, those that were identified as low risk had a refusal 
rate of just 4.7%, while high-risk cases had a refusal rate of 24.6%. 


Refusal Rate by Risk Level 


Average refusal 
rote: 8.838 


Low Risk Medium Risk High Risk 


‘re working toward a pilot for Overseas spousal sponsorships 
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.15(1)(i)i) 


Predictive models can be used to 
strengthen program integrity by 
identifying high-risk cases. 


For eTA of nationals, 
preliminary results suggest that a 
predictive model detects more than 
five times as many high-risk clients 
compared to quality assurance done 
through random selection. 


Better targeting can increase the 
“hit rate” and reduce the amount of 
resources dedicated to screening. 
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Example 4: Flagging High Risk Cases for € 


Percentage of high-risk eTA cases 
detected, by targeting approach 


30 
26.0 
25 


Random checks Predictive modei 


My last example pertains to ETA. As you're probably aware, was a big day 


for ETA hecause 
visit Canada, 


became 


{o 


PIB has asked us to work with them to identify 
indicators of high-risk applications that should crop out of the automated process and be 
manually reviewed, 


Based on very preliminary work, if we used a model to pick high risk applications, about 
iin 4 would be from a client for whom we would refuse entry inte Canada. Thats much 
higher than if we just picked applications randomly, in which case less than 5% would be 


refused entry. 
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Business Rules and Predictive Analytics 


Business rules can be a great tool, but they are not the same as predictive analytics. 


Suppose we're trying to identify low-risk applications: 


Business Rules 


Business rules are increasing y being used To expedite processing. 
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A well designed business rule can be a great tool. But ivs not the same as predictive 


analytics. 
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Pros and Cons of Predictive Analytics Compared to Busine 


: Advantages 


+ Exhaustive. Models consider how all available * More complex. Predictive analytics requires 
information correlates with the outcome. Business specialized skills and software. 


rules are limited by our imagination. : 
+ Higher upfront cost. Models take more time and 


Granularity, Models create rules with many resources to create and deploy. However: 
conditions that target very similar cases. 

~ {T infrastructure required for predictive models 
Flexibility. Business rules that are hard-coded into a 
system cannot be updated regularly; models can be 
updated more easily. 


* informative. Models use probability scores to 
specify each the degree of risk, rather than just a 
“pass” or “fail. 


Exhaustive: Models consider how all available information correlates 
with the outcome. it draws on the main strength of computers, W 
is their ability to do millions of calculations per second. The cor 
will try all the combinations of applicant characteristics and see ho: 
they correlate with the predicted outcome. Business rules are limited by 
our imagination. 


ranularity: 
e Models create rules with many conditions that target very similar 
cases. 
* The granularity offers potential for greater precision in targeti 
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What is Predictive Analytics? 


Generally speaking, predictive analytics uses computers to analyze past behaviors 
to discover patterns that enable predictions of future behaviours. 


Predictive analytics 
draws upon a 
combination of 


k 


* Advanced Statistics 
* Computer Science 
+ Business Expertise 


+ Specialized Software 


Using these assets, 
trends within data sets 
can be uncovered and 

linked ta outcomes. 


An evidence-based 
decision making mode! 
that substantiates 
dacisions made by 
human intuition. 
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Predictive Analytics Is Being Increasingly 


Predictive analytics is being successfully used in the private, public, and non-profit 
sectors to enhance how organizations serve their clients. Some examples: 
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Vancouver Police Department F p 
Medway Youth Tru 
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Predictive Analytics at IRCC 


At iRCC, predictive models are being developed to: 


— Triage applications between high, medium and low risk. 
— Support fraud detection. 


A model is built by analyzing thousands of past applications, along with almost all other 
information available to officers at the time a decision is made. 


The computer analyzes how combinations of applicant characteristics correlate with 
approval, refusal, fraud, etc. 


The strongest and most convincing correlations are used to inform decision making. 


At IRCC, predictive models are being developed to: 


* Rather than just identify low-risk cases, we could triage all 
applications between high, medium and low risk. 


e Support fraud detection. We’re currently working on this for CIT 


an: 


A model is built by analyzing thousands of past applications, along 
with almost all other information available to officers at the time a 
decision is made. 


The computer analyzes how combinations of applicant 
characteristics correlate with approval, refusal, fraud, etc. 


The strongest and most convincing correlations are used to inform 
decision making. 


A-2018-09320-000394 


Local Intelligence is Critical 


Predictive models are not a substitute for local expertise; the two are complementary. 


Models are much more accurate when enriched by local patterns identified by officers. 
For example: 


it will be essential for model developers to work closely with 
PIB, IN, CN and DN to have the most updated intelligence. 
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Benefits of Predictive Analytics at IRCC 


Officers can be freed up to: 
~ Focus on complex cases 
— Strengthen program integrity 


~ Reduce backlogs 


Models can support program integrity directly by 
flagging high-risk cases for greater scrutiny. 


Focus on complex cases 
Strengthen program integrity 
Reduce backlogs 


Models can also flag the highest-risk cases that merit extra scrutiny. I'll 
show you an example of what we’re doing in this area. 


In the future, we aim to predict undesirable behaviours (e.g. criminality 
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* A one-day experiment will be conducted on December 16, 2016. 


$ 


«  Aone-day field trial is planned for December 16, 2016. 


Immigration, Refugees Immigration, Réfugiės 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


s.16(1)(b) 
s.16(2) 


We've already done many simulations with this model and some dress rehearsals of 


what would happen on the one-day experiment. 
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Example 2: Risk Triage of Spousal Sponsorships 


in June 2016, predictive models were used to perform a risk triage of the entire in-Canada 
spousal inventory. 


Each application was assigned a rating of high, medium or low risk, to assist officers when 
deciding how much scrutiny may be required. 


Among finalized applications since June, those that were identified as low risk had a refusal 
rate of just 4.7%, while high-risk cases had a refusal rate of 24.6%. 


Refusal Rate by Risk Level 


Average refusal 
rote: 8.838 


Low Risk Medium Risk High Risk 


‘ve been tracking the final decisions on the inventory to see how 
accurate the model’s triage was. 
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Over the past year, we've worked with PIB on a proof of concept to determine if 
precictive analytics can help us find risk Indicators for Passport identity fraud. 


The proof of concept was successful. The model was able to identify pockets of nigh risk 
people and sockets of very low risk people. 


Here is an example of a risk indicator for high-risk applicants... 


These findings are preliminary; this is just a proof of concept, so we're not 
recommending that this indicator be used in the field, but it gives an idea of the kind of 
atterns that a el can recognize that might not jump out to the naked eye. 


Wer ings thing similar for CITZ 
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s.15(1)(i)i) 


Predictive models can be used to 
strengthen program integrity by 
identifying high-risk cases. 


For eTA of rationals, 
preliminary results suggest that a 
predictive model detects more than 
five times as many high-risk clients 
compared to quality assurance done 
through random selection. 


Better targeting can increase the 
“hit rate” and reduce the amount of 
resources dedicated to screening. 
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Example 4: Flagging High Risk Cases for € 


Percentage of high-risk eTA cases 
detected, by targeting approach 


30 
26.0 
25 


Random checks Predictive modei 


My last example pertains to ETA. As you're probably aware, was à big day 


for ETA hecause 
visit Canada, 


became 


{o 


PIB has asked us to work with them to identify 
indicators of high-risk applications that should crop out of the automated process and be 
manually reviewed, 


Based on very preliminary work, if we used a model to pick high risk applications, about 
iin 4 would be from a client for whom we would refuse entry inte Canada. Thats much 
higher than if we just picked applications randomly, in which case less than 5% would be 


refused entry. 
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Business Rules and Predictive Analytics 


Business rules can be a great tool, but they are not the same as predictive analytics. 


Suppose we're trying to identify low-risk applications: 


Business Rules 


Business rules are Increasingly being used to expedite processing. 


A wel designed business rule can be a great tool. But ivs not the same as predictive 
analytics. 
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Pros and Cons of Predictive Analytics Compared to Busine 


: Advantages 


+ Exhaustive. Models consider how all available * More complex. Predictive analytics requires 
information correlates with the outcome. Business specialized skills and software. 


rules are limited by our imagination. : 
+ Higher upfront cost. Models take more time and 


Granularity, Models create rules with many resources to create and deploy. However: 
conditions that target very similar cases. 

~ {T infrastructure required for predictive models 
Flexibility. Business rules that are hard-coded into a 
system cannot be updated regularly; models can be 
updated more easily. 


* informative. Models use probability scores to 
specify each the degree of risk, rather than just a 
“pass” or “fail. 


Exhaustive: Models consider how all available information correlates 
with the outcome. it draws on the main strength of computers, W 
is their ability to do millions of calculations per second. The cor 
will try all the combinations of applicant characteristics and see ho: 
they correlate with the predicted outcome. Business rules are limited by 
our imagination. 


ranularity: 
e Models create rules with many conditions that target very similar 
cases. 
* The granularity offers potential for greater precision in targeti 
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Our Next Steps 
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Roll out new pilot projects to test 
additional models in the real world. 


Develop new predictive models for 
other lines of business. 


Work with the SIMB to design the IT 
systems required for deployment. 


Seek funding for building these IT 
systems. 


Keep integrating cutting-edge 
predictive analytics techniques. 
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For more information... 


Steven Gonzalez 
Assistant Director, Predictive Analytics Team 
Business Research, Intelligence and Analytics Directorate (OPPB) 


Steven. Gonzalez@cic.gr.ca 
613-437-5556 


At IRCC, predictive models are being developed to: 


* Rather than just identify low-risk cases, we could triage all 
applications between high, medium and low risk. 


rking on this for CIT 


e Support fraud detection. We're currently 


A model is built by analyzing thousands of past applications, along 
with almost all other information available to officers at the time a 
decision is made. 


The computer analyzes how combinations of applicant 
characteristics correlate with approval, refusal, fraud, etc. 


The strongest and most convincing correlations are used to inform 
decision making. 
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Immigration, Refugees immigration, Réfugiés 
and Citizenship Canada et Citoyenneté Canada 


Using Predictive 
Analytics to Reduce 
Processing Times 


April 20, 2016 


Canada 
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Predictive analytics is an innovative technique that consists of using computers to 
analyze past behaviours. Through this analysis, we can discover patterns that 
enable predictions of future actions. 


— 


e Statistics e Supports evidence-based decision making; 


e Computer Science e Substantiates decisions made by intuition; 


e Business Expertise e Enables faster business processes; 


e Specialized Software 


e Helps maximize resource output; 


e Increases consistency in decision making; 


e Generates synergies between business 
expertise and statistical analysis. 
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e In 2016, IRCC will receive more than 2 million applications from people wanting to 
come to Canada to study, work, or visit. For each application, officers must decide 
whether to approve or refuse it. 


e The Data and Predictive Analytics Team (DPAT) is building predictive models designed 
to mimic these decisions about whether an application should be approved. 


e  DPAT works with program experts and stakeholders to understand the decision-making 
process, and the importance of each variable on the final decision 


e To discuss best practices, DPAT has established relationships with other countries that 
are currently using predictive analytics 
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The application decision making process today... 


Application 


entered in 
arrives at IRCC 


system 


Application 


Application 
checked for 


completeness 
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Applications 
reviewed by 
humans for decision 


And with predictive analytics... 


Application 
entered in 
system 


Application 


arrives at IRCC 
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Application 
checked for 
completeness 


Canada -+ 
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e This innovative approach frees up officers to: 


— Reduce any application backlogs; 
— Focus on complex cases, and; 
— Strengthen program integrity. 


e Future uses of predictive analytics: 


— Predict outcomes (e.g., who is likely to 
commit a crime in Canada). 

— Improve workload distribution by 
automatically assigning cases to the 
appropriate officers. 
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Volume of Temporary Resident Visa (TRV) Applications in 2015 
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Volume of Temporary Resident Visa (TRV) Applications in 2015 
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The model reached the same decision as 
officers in 98.3% of 
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e Design and build IT systems; 


e Test existing models in real time, 
once the necessary IT systems are 
implemented; 


e Develop new predictive models for 
other lines of business; 


e Develop models to predict outcomes 
(e.g. criminal activity); 


e Develop models for workload 
distribution; 


° Keep integrating cutting edge 
predictive analytics techniques. 


| 
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What is Predictive Analytics? 


Generally speaking, predictive analytics uses computers to analyze past behaviors 
to discover patterns that enable predictions of future behaviours. 


Predictive analytics 
draws upon a 
combination of 


k 


* Advanced Statistics 
* Computer Science 
+ Business Expertise 


+ Specialized Software 


Using these assets, 
trends within data sets 
can be uncovered and 

linked ta outcomes. 


An evidence-based 
decision making mode! 
that substantiates 
dacisions made by 
human intuition. 
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Predictive Analytics Is Being Increasingly 


Predictive analytics is being successfully used in the private, public, and non-profit 
sectors to enhance how organizations serve their clients. Some examples: 
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Predictive Analytics at IRCC 


At iRCC, predictive models are being developed to: 


— Triage applications between high, medium and low risk. 
— Support fraud detection. 


A model is built by analyzing thousands of past applications, along with almost all other 
information available to officers at the time a decision is made. 


The computer analyzes how combinations of applicant characteristics correlate with 
approval, refusal, fraud, etc. 


The strongest and most convincing correlations are used to inform decision making. 


At IRCC, predictive models are being developed to: 


Rather than just identify low-risk cases, we could triage all 
applications between high, medium and low risk. 


Support fraud detection. We’re currently working on this for CIT 


an: 


A model is built by analyzing thousands of past applications, along 
with almost all other information available to officers at the time a 
decision is made. 


The computer analyzes how combinations of applicant 
characteristics correlate with approval, refusal, fraud, etc. 


The strongest and most convincing correlations are used to inform 
decision making. 
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Benefits of Predictive Analytics at IRCC 


Officers can be freed up to: 
— Focus on complex cases 
~ Strengthen program integrity 
— Reduce backlogs 


Models can support program integrity directly by 
flagging high-risk cases for greater scrutiny. 


in the future, we will try to predict undesirable 
behaviour (e.g. criminality} 


Focus on complex cases 
Strengthen program integrity 
Reduce backlogs 


Models can also flag the highest-risk cases that merit extra scrutiny. I'll 
show you an example of what we’re doing in this area. 


In the future, we aim to predict undesirable behaviours (e.g. criminality 
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* A one-day field trial is planned for December 16, 2016. 


« Simulations to date: 


* A one-day experiment will be conducted on December 16, 2016. 


* We've already done many simulations with this model and some dress rehearsals of 
what would happen on the one-day experiment. 
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Example 2: Risk Triage of Spousal Sponsorships 


in June 2016, predictive models were used to perform a risk triage of the entire in-Canada 
spousal inventory. 


Each application was assigned a rating of high, medium or low risk, to assist officers when 
deciding how much scrutiny may be required. 


Among finalized applications since June, those that were identified as low risk had a refusal 
rate of just 4.7%, while high-risk cases had a refusal rate of 24.6%. 


Refusal Rate by Risk Level 


Average refusal 
rote: 8.838 


Low Risk Medium Risk High Risk 


‘ve been tracking the final decisions on the inventory to see how 
accurate the model’s triage was. 
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Over the past year, we've worked with PID on a proof of concept to determine if 
precictive analytics can help us find risk Indicators for Passport identity fraud. 


The proof of concent was successful. The model was able to identify pockets of nigh risk 
people and sockets of very low risk people. 


Here is an example of a risk indicator for high-risk applicants... 


These findings are preliminary; this is just a proof of concept, so we're not 
recommending that this indicator be used in the field, but it gives an idea of the kind of 
atterns that a el can recognize that might not jump out to the naked eye. 


Wer ings thing similar for CITZ 
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Example 4: Flagging High Risk Cases for € 


Predictive models can be used to 
strengthen program integrity by 


identifying high-risk cases. Percentage of high-risk eTA cases 


detected, by targeting approach 


For eTA of nationals, 30 

preliminary results suggest that a 26.0 
predictive model detects more than 25 

five times as many high-risk clients 

compared to quality assurance done 

through random selection. 


Better targeting can increase the 


“hit rate” and reduce the amount of 
resources dedicated to screening. 


Random checks Predictive modei 


My last example pertains to ETA. As you're probably aware, was a big day 
for ETA because became 
visit Canada. 


PIB has asked us to work with them to identify 
indicators of high-risk applications that should crop out of the automated process and be 
manually reviewed, 


Based on very preliminary work, if we used a model to pick high risk applications, about 
iin 4 would be from a client for whom we would refuse entry inte Canada. That’s much 
higher than if we just picked applications randomly, in which case less than 5% would be 
refused entry. 


A-2018-09320-000426 


Business Rules and Predictive Analytics 


Business rules can be a great tool, but they are not the same as predictive analytics. 


Suppose we're trying to identify low-risk applications: 


Business Rules 


Business rules are increasing y being usec To expedite processing. 
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A well designed business rule can be a great tool. But ivs not the same as predictive 


analytics. 
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Pros and Cons of Predictive Analytics Compared to Busine 


: Advantages 


+ Exhaustive. Models consider how all available * More complex. Predictive analytics requires 
information correlates with the outcome. Business specialized skills and software. 


rules are limited by our imagination. : 
+ Higher upfront cost. Models take more time and 


Granularity, Models create rules with many resources to create and deploy. However: 
conditions that target very similar cases. 

~ {T infrastructure required for predictive models 
Flexibility. Business rules that are hard-coded into a 
system cannot be updated regularly; models can be 
updated more easily. 


* informative. Models use probability scores to 
specify each the degree of risk, rather than just a 
“pass” or “fail. 


Exhaustive: Models consider how all available information correlates 
with the outcome. it draws on the main strength of computers, W 
is their ability to do millions of calculations per second. The cor 
will try all the combinations of applicant characteristics and see ho: 
they correlate with the predicted outcome. Business rules are limited by 
our imagination. 


ranularity: 
e Models create rules with many conditions that target very similar 
cases. 
* The granularity offers potential for greater precision in targeti 
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Our Next Steps 


Roll out new pilot projects to test 
additional models in the real world. 


Develop new predictive models for other 
lines of business. 


Design models te predict specific outcomes 
(e.g. criminal activity). 


Work with the SIMB to design the IT 
systems required for deployment. 


Seek funding for building these IT systems, 


Keep integrating cutting-edge predictive 
analytics techniques. 
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For more information... 


Steven Gonzalez 
Assistant Director, Predictive Analytics Team 
Business Research, Intelligence and Analytics Directorate (OPPB) 


Steven. Gonzalez@cic.gr.ca 
613-437-5556 


At IRCC, predictive models are being developed to: 


* Rather than just identify low-risk cases, we could triage all 
applications between high, medium and low risk. 


rking on this for CIT 


e Support fraud detection. We're currently 


A model is built by analyzing thousands of past applications, along 
with almost all other information available to officers at the time a 
decision is made. 


The computer analyzes how combinations of applicant 
characteristics correlate with approval, refusal, fraud, etc. 


The strongest and most convincing correlations are used to inform 
decision making. 
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What is Predictive Analytics? 


Generally speaking, predictive analytics uses computers to analyze past behaviors 
to discover patterns that enable predictions of future behaviours. 


Predictive analytics 
draws upon a 
combination of 


k 


* Advanced Statistics 
* Computer Science 
+ Business Expertise 


+ Specialized Software 


Using these assets, 
trends within data sets 
can be uncovered and 

linked ta outcomes. 


An evidence-based 
decision making mode! 
that substantiates 
dacisions made by 
human intuition. 
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Predictive Analytics Is Being Increasingly 


Predictive analytics is being successfully used in the private, public, and non-profit 
sectors to enhance how organizations serve their clients. Some examples: 


é 
= 


ie ne 
Ne 


t 


3 


Vancouver Police Department F p 
Medway Youth Tru 
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Predictive Analytics at IRCC 


At iRCC, predictive models are being developed to: 


— Triage applications between high, medium and low risk. 
— Support fraud detection. 


A model is built by analyzing thousands of past applications, along with most of the 
information available to officers at the time a decision is made. 


The computer analyzes how combinations of applicant characteristics correlate with 
approval, refusal, fraud, etc. 


The strongest and most convincing correlations are used to inform decision making. 


At IRCC, predictive models are being developed to: 


* Rather than just identify low-risk cases, we can triage all applications 
between high, medium and low risk. 


e Support fraud detection. We’re currently working on this for CIT and PP. 


A model is built by analyzing thousands of past applications, along with most 
of the information available to officers at the time a decision is made. 


The computer analyzes how combinations of applicant characteristics 
correlate with approval, refusal, fraud, etc. 


The strongest and most convincing correlations are used to inform decision 
making. The best correlations are tested extensively on data that the model 
has never seen, to determine if the model’s insights are reliable and can be 
generalized to other cases. 
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Benefits of Predictive Analytics at IRCC 


— Focus on complex cases 
— Strengthen program integrity 


— Reduce backlogs 


Flagging high-risk cases supports program 
integrity more directly by making sure that these 
cases do not fall through the cracks. 


We are beginning to predict undesirable 
behaviour (e.g. removal orders} 


Focus on complex cases 
Strengthen program integrity 
Reduce backlogs 


Models can also flag the highest-risk cases that merit extra scrutiny. 


We are beginning to predict undesirable behaviours (e.g. removal 


orders) 
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À one-day pilot was conducted on December 16, 
2016. 


— The quality of the model decisions seems good 
(avoiding adverse outcomes) 


DM and DMA were very pleased with results, 


ADM Orr has asked for implementation before the 
2018 peak season. 


The DM and DMA were briefed on the results. They were very pleased 
and would like us to press forward on this. 
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The ADM of Operations has asked for implementation before the 2018 
peak season for China. 
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The pilot was for applications received via a VAC. However, IN has requested that we focus 
on e-apps, so the rest of the discussion focusses on our new model for TRVs received via e- 


apps 


Database: Jan 2015 to February 2017 
According to section 42 of IRPA, if a person is inadmissible to Canada, their family members 
could also be inadmissible. 
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Exemple 1: Data Sources for China Model 


Consent clause on application for temporary resident visa (TRV): 
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The model finds people that have very similar characteristics and very similar outcomes in 
terms of decisions from our officers. 
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Related parties: includes famiy members, other people traveling with the cllent as a group, anc the 

inviter. 

* You may recall that section 42 of IRPA states that a person may be inadmissible into Canada if one of 
their family members is inadmissible. 
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À predictive model was built to identify low-risk spousal sponsorships that could be granted 
streamlined processing. 


The data used to huild the model covered roughly 100,000 applications. 


The data used is similar to the China model, but we have information about both the sponsor 
and the principal applicant: 


* Allof this information is already collected under current operating procedures and is already 
used by officers when adjudicating applications. 


A-2018-09320-000442 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


$.16(1)(b) 
s.16(2) 


Example 2: Risk Triage of Spousal Sponsorships 


+ The model generated many rules grouping very similar applicants, as in the China example. 


* ‘his triage rule is currently being piloted in Manila and New Delhi. 


Philippines and India are among the lower-risk countries for this line of 
business, among visa-required countries. We also wanted to limit the 
pilot to a small number of offices. 
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Data Governance and Legal Risks 


Section 6{2) of the Privacy Act requires that all reasonable steps be taken to ensure that 
personal information used for administrative decisions is as accurate, up-to-date and 
complete as possible. 

This presupposed a certain quality of data in GCMS and other systems. 


We know that data quality problems already exist at IRCC 


# This could create legal risks for current manual processing, before predictive analytics 
is ever deployed. 


OPP has a Data Governance Team dedicated to working with branches acrass IRCC to 
ensure high quality data. 


This work deserves greater emphasis regardiess of the future of predictive analytics. 


But | want to make sure that we con't lose sight of a legal consideration that pre-dates 
precictive analytics and affects how IRCC currently does business. Irn talking about Data 
Governance or data quality. 


Example: duplicate UCE to qualify for some programs, you need experience with another 


program. CEC requires previous work experience in Canada, hence a WP. lf the WP is on 
one UCI and the CEC application on another, the person may not appear eligible, 
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Next Steps for Predictive Analytics 


Continue developing models identifying 
low-risk applications 


* Focus on TRY, student permits and 
spousal sponsorships 


Continue developing models to identify 
high-risk cases 


+ Focus on eTA, Citizenship and 
Passport 


Anew China TRV pilot is forthcoming; 
india TRV to follow. 


Work with SIMB te depley China model 
by February 2018; India to follow later in 
2018. 
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For more information... 


Steven Gonzalez 
Assistant Director, Predictive Analytics Team 
Business Research, Intelligence and Analytics Directorate (OPPB) 


Steven. Gonzalez@cic.gr.ca 
613-437-5556 


At IRCC, predictive models are being developed to: 


* Rather than just identify low-risk cases, we could triage all 
applications between high, medium and low risk. 


rking on this for CIT 


e Support fraud detection. We're currently 


A model is built by analyzing thousands of past applications, along 
with almost all other information available to officers at the time a 
decision is made. 


The computer analyzes how combinations of applicant 
characteristics correlate with approval, refusal, fraud, etc. 


The strongest and most convincing correlations are used to inform 
decision making. 
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Business Rules and Predictive Analytics 


Business rules can be a great tool, but they are not the same as predictive analytics. 


Business Rules 


Business rules are increasingly being used to expedite processing. 


A wel designed business rule can be a great tool But it’s not the same as predictive analytics. 
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Pros and Cons of Predictive Analytics Compared to Busine 


: Advantages 


+ Exhaustive. Models consider how all available * More complex. Predictive analytics requires 
information correlates with the outcome. Business specialized skills and software. 


rules are limited by our imagination. : 
+ Higher upfront cost. Models take more time and 


Granularity, Models create rules with many resources to create and deploy. However: 
conditions that target very similar cases. 

~ {T infrastructure required for predictive models 
Flexibility. Business rules that are hard-coded into a 
system cannot be updated regularly; models can be 
updated more easily. 


* informative. Models use probability scores to 
specify each the degree of risk, rather than just a 
“pass” or “fail. 


Exhaustive: Models consider how all available information correlates 
with the outcome. it draws on the main strength of computers, W 
is their ability to do millions of calculations per second. The cor 
will try all the combinations of applicant characteristics and see ho: 
they correlate with the predicted outcome. Business rules are limited by 
our imagination. 


ranularity: 
e Models create rules with many conditions that target very similar 
cases. 
* The granularity offers potential for greater precision in targeti 
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Predictive Analytics for ETA 
Business Context 


Pick out low risk travellers from visa required countries; allow them to apply for an eTA (electronic travel 
authorization) instead of a TRV (temporary resident visa). 


Eligibility: candidate was previously issued a Canadian TRV within the last ten years; or holds a valid U.S. 
non-immigrant visa 


Pilot project for low risk travellers in 2015; will be rolled out globally eventually 


Lab data available: 324,530 TRV applications from citizens, between 2008 and 2012 
36 fields for each record 


Data cleansing: deal with missing values and outliers; remove fields that have a high percentage of 
unknown data 


Modeling 


Decision tree 


Chart? 

How to pilot the model 

How the model can be tested in the real world in a zero risk fashion 
What results are we looking for in order to improve the model? 
Deployment plans 

Integration with application (GCMS)? 

Change management — what business processes will change? 


Risk management — if something goes wrong 
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Collet bot ond bee aeons 


T onhxt preliminary Date Analy 


Concrete benefits in any business doing predictive analytics (Walmart, for example, or 
Googles, that both use that now in their business to save money, profile their consumers and 
improve resource allocations) 


The benefits of combining operations expertise with empirical data and statistical analysis to 
buld a predictive model 


e increasing processing capacity 


+ Facilitate program integrity exercises 
+ Deliver optimal client service 


Data Analytics: out reach 


Data Analytics Working Groups 
predictive analytics demands tight collaboration with partners and the business 
We started the WG in collaboration with PID. We co-chair. 


Data Analytics Coalition 

Great forum for sharing ideas and best practices as predictive analytics is relatively new to the 
federal government 

Similar goals — workload distribution 

Common challenges — change management, setting users expectations, poor data quality 


2 
A-2018-09320-000452 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


Express Entry- Data Exploration 


“Original data set had 547,215 applications with 122 unique fields 


Federal Skilled Worker Program (FSW) - 72.13% 
Provincial/Territorial Nominees (P/N) - 21.16% 
Canadian Experience Class (CEC) - 6.69% 
Skilled Trades {ST} - 0.03% 
Distribution by Programs 
Sie Workers 
FrovincialTenitonal Nominees] 
Canadian Experience Class Li 
Suited Trades! 


« Missing values and outliers were removed from the data for each LOB 


+ As a result, each LOB has reduced number of records 
* Time periods for modeling were also chosen for each LOB depending on availability 
and quality of the data 
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Express Entry- Data Used (conti 


Total number of valid recards consist of 155,665 anciications 
Tatai number of fields is 122 unique fields 
* Dates covered (final decision dates} are from 2007-01-04 te 2008-02-27 


«+ Total number of valid records consist of 111,060 applicatinns 
+ Total number of fields is 122 unique fields 
+ Dates covered {final decision datesi are from 2007-01-02 to 2014-08-18 


+ Total number of valid records consist of 25,058 appiications {GOMMS only} 
+ Total number of fields is 122 unique fielde 
+ Dates covered (fing! decision dates} are fryer: 2011-07-16 to 2014-08-15. 


For FSW, DPAT analyzed data for FSW before and after MI. As a result, before MI data 
was chosen for FSW. This decision was based on business knowledge and opinion on 
which cases should be included in the sample in the way that there would not be any 
lost in representativeness of our future population of applications, while preserving the 
quality of the data set. The fact that Express Entry does not have MI restrictions is also 
consistent with this choice of pre-Ml time period for modeling. 


For PN, data was analyzed for GCMS and CAIPS separately due to some missing values 
problem. since both 
sets of data yield the same acceptance rate and adding more applications to build the 
model results in a higher degree of freedom leading to increased robustness. Thus 
helping to ensure that the model performs as expected (i.e. is not over-fitted). 


For CEC, WG suggested that DPAT use only GCMS data from 2011 to 2014-07-04 (final 
decision date) given all the missing values in CAIPS. 
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DPAT used Auto data Preparation methot 
to identify the most important predictors 
for making decision to approve or refuse 
the application. 


DPAT bave looked at different predictior 
methods, such as Decision Trees {Le 
C5,6, CHIAD...) and Logistic Regression. 


After developing models based on each 
al these methodologies, we compare the 
averall accuracy rate, discuss the result: 
with the clents and select the methoc 
with the highest possible accuracy rate 
that meets CIC’s needs. 


It is important to focus our modeling on the fields/indicators that matter the most and 
consider dropping those that matter least. Predictor importance does not relate to 
model accuracy. It chooses among available 122 fields 


The predictive power (non-weighted) of each recommended predictor is computed by 
Auto Data Preparation (ADP) from either a linear regression or naive Bayes model, 
depending upon whether the target is continuous or categorical. In our case, the target 
FinDec is categorical so that Bayes naive model was used by ADP. 


The Bayes model is termed naive because it treats all proposed prediction variables as 
being independent of one another. This is why we used naive Bayes model only for 
initial selection of predictors. 
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Fach Express Entry application coming trough will be assessed and scored against 
predictors and as per the mode! that was developed and will be re-validated before 


implementation, 


The predictive model breaks down the Express Entry casedoad into 3 Bins: 


5 inian decision maten 


Cases in the Green Bin and Red Sin {applications scored with most certainty) will go to in 


Level ane and Level two decision makers; 


Cases in Yellow Bin (cases with some uncertainty} will go to Level three decision makers. 


Canada 


The probability bins like exam marks. Red and Green bins are passing marks for 
applications to be processed in Canada. There will be certain number of applications 
that pass this mark. This number of applications depends on the threshold (in our case 
85% for PN and CEC and 80% for FSW) that is established in agreement with business 
and our partners. 


For CEC, 99% applications is 
processed in Canada so that cases corresponding to Yellow bin will go for the additional 
QA. 


For FWS, 
PN, CEC was used C5.0. algorithm to develop the models. Each of these LOB has 
different output for probability bins. 
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For FSW, 


For PN, 


For CEC, 


7 
A-2018-09320-000457 


Immigration, Refugees Immigration, Réfugiés 
| + | and cistet et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulge (2} de la loi sur l'accès à l'information 


For FSW, 
There were 


39.2% of cases to be referenced since the probability P for FinDec to be correct was 
less than 80% or the prediction was biased. 


For PN, 
There were 


31.2% of cases to be referenced since the probability P for FinDec to be correct was 
less than 80% or the prediction was biased. 


For CEC, 

There were 
3.4% of cases to be referenced since the probability P for FinDec to be correct was less 
than 80% or the prediction was biased. It is expected that business and our partners 
will tell us what the future plans are for the split in processing for CEC. 
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Express Entry ~ Next Steps 


Phase |. Model results are used to produce traping niles for design of 
Express Entry 


More work needs to be done in the next few months to improve the 


For Phase I, this is about November, 2014 


*For Phase Il, this is about 6-9 months after launching Express Entry in January, 2015 
when Express Entry data will be available 


*Then it will be ongoing maintenance (monitoring, revision if needed) of the model 
Details: 

elt is anticipated to restart WG after receiving Express Entry data. 

*The current triage rules and risk profiles are based on the subjective input of visa 


officers. 
The future plan is to make these rules exhaustive and evidence-based. 
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Collet bot ond bee aeons 


T onhxt preliminary Date Analy 


Concrete benefits in any business doing predictive analytics (Walmart, for example, or 
Googles, that both use that now in their business to save money, profile their consumers and 
improve resource allocations) 


The benefits of combining operations expertise with empirical data and statistical analysis to 
buld a predictive model 


* increasing processing capacity 


+ Facilitate program integrity exercises 
+ Deliver optimal client service 


Data Analytics: out reach 


Data Analytics Working Groups 
predictive analytics demands tight collaboration with partners and the business 
We started the WG in collaboration with PID. We co-chair. 


Data Analytics Coalition 

Great forum for sharing ideas and best practices as predictive analytics is relatively new to the 
federal government 

Similar goals — workload distribution 

Common challenges — change management, setting users expectations, poor data quality 
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Express Entry- Data Exploration 


“Original data set had 547,215 applications with 122 unique fields 


Federal Skilled Worker Program (FSW) - 72.13% 
Provincial/Territorial Nominees (P/N) - 21.16% 
Canadian Experience Class (CEC) - 6.69% 
Skilled Trades {ST} - 0.03% 
Distribution by Programs 
Sie Workers 
FrovincialTenitonal Nominees] 
Canadian Experience Class Li 
Suited Trades! 


« Missing values and outliers were removed from the data for each LOB 


+ As a result, each LOB has reduced number of records 
* Time periods for modeling were also chosen for each LOB depending on availability 
and quality of the data 
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Express Entry- Data Used (conti 
Total number of valid recards consist of 155,665 anciications 
Tatal number of fields is 122 unique fields 

+ Dates covered (final decision dates} are from 2007-01-04 te 2008-02-27 

* Total number of valid records consist of 111,060 applicatinns 

+ Total number of fields is 122 unique fields 

» Dates covered (fing! decision datés are from 2007-01-02 to 2014-08-18 

* Total number of valid records consist of 25,058 applications (GCAMS only} 

+ Total number of fields is 122 unique fielde 

+ Dates covered (fing! decision dates} are fryer: 2011-07-16 to 2014-08-15. 
For FSW, DPAT analyzed data for FSW before and after MI. As a result, before MI data 
was chosen for FSW. This decision was based on business knowledge and opinion on 
which cases should be included in the sample in the way that there would not be any 
lost in representativeness of our future population of applications, while preserving the 
quality of the data set. The fact that Express Entry does not have MI restrictions is also 
consistent with this choice of pre-MI time period for modeling. 
For PN, data was analyzed for GCMS and CAIPS separately due to some missing values 
problem. since both 
sets of data yield the same acceptance rate and adding more applications to build the 
model results in a higher degree of freedom leading to increased robustness. Thus 
helping to ensure that the model performs as expected (i.e. is not over-fitted). 
For CEC, WG suggested that DPAT use only GCMS data from 2011 to 2014-07-04 (final 
decision date) given all the missing values in CAIPS. 
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DPAT used Auto data Preparation method 
to identify the most important predictors 
for making decision to approve or refuse 
the application. 
DPAT have looked at different prediction 
methods, such as Decision Trees {Le. 
C30, CHIAD...) and Logistic Regression. 
After developing models based on each 
afl these methodologies, we compare the 
overall accuracy rate, discuss the results 
with the clents and select the method 
with the highest possible accuracy rate 
that meets CIC’s needs. 
It is important to focus our modeling on the fields/indicators that matter the most and 
consider dropping those that matter least. Predictor importance does not relate to 
model accuracy. 
The predictive power (non-weighted) of each recommended predictor is computed by 
Auto Data Preparation (ADP) from either a linear regression or naive Bayes model, 
depending upon whether the target is continuous or categorical. In our case, the target 
FinDec is categorical so that Bayes naive model was used by ADP. 
The Bayes model is termed naïve because it treats all proposed prediction variables as 
being independent of one another. This is why we used naive Bayes model only for 
initial selection of predictors. 
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Scores and models are not evervthing: 


Ine placement oT tne business rules mignt change atter we have come up with a Solution (IT) as this is just 
tentative for now. 


Complexity will be added in the future to the Predictive Models, in order to do complete complexity and risk- 
based triaging. 


Quality Assurance is a random selection of XX% of cases for QA, assigned to QA group. 
Workloads Distribution: Each bin can be assigned to officers according to operational requirements, expertise, levels, etc. 


Triage Model Validation: A random selection of cases from bin #1 and # 2 will be assigned to officers at superior levels to ensure 
that: 


Higher ranking officers remain exposed to all types of case. 

Senior officers can continuously validate the Triage model. 

Experienced officers can confirming that cases were effectively assigned to the right bin by the Triage model. 
Experienced field officers can detect errors in the Triage model. 


Experienced field officers can provide intelligence to centralised offices and HQ in order to improve and rafine the Triage model. 


Note: 


For FSW, PN, CEL was used LD.U. algorithm to 
develop the models. Each of these LOB has different output for probability bins. 
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Express Entry ~ Next Steps 


Phase |. Model results are used to produce traping niles for design of 
Express Entry 


More work needs to be done in the next few months to improve the 


For Phase I, this is about November, 2014 


*For Phase Il, this is about 6-9 months after launching Express Entry in January, 2015 
when Express Entry data will be available 


*Then it will be ongoing maintenance (monitoring, revision if needed) of the model 
Details: 

elt is anticipated to restart WG after receiving Express Entry data. 

The current triage rules and risk profiles are based on the subjective input of visa 


officers. 
The future plan is to make these rules exhaustive and evidence-based. 
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OVERVIEW: IRCC Predictive Analytics Projects 


Senior management approved $10M over 3 years to build and deploy predictive analytics models and 
forecasting models across several lines of business. 


i. Deploy predictive models to support processing of visitor visas and student permits far China and 
india nationals (submitted online}, as a first step towards a global salution: 


a, Daily risk triage of applications. 
b. 


Predictive models te support risk-triage and fraud detection far Citizenship and Passport applications. 


Develop forecasting models for several lines af business (visitor visas, student permits, work permits, 
Citizenship Grants, Permanent Resident Cards}. 


Develop a Business intelligence {Bi} and Exploratory Environment (i.e. Advanced Analytics sandbox) to 
support the project. 


Other Predictive Modeling Projects 
i. Proof of concent to identify high-risk applicants and trends for electronic travel authorizations (eTA). 
Classification models to triage and centralize processing of low-risk spousal sponsorships. 


Assessing IRCC client service performance (sentiment analysis) 
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CASE STUDY 


CHINA TEMPORARY RESIDENT VISA DEPLOYMENT (2018) 
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Deploying Predictive Models: Temporary Resident Visas 


* Amodel analyzes thousands of past officer decisions, along with most of the information 
available to officers at the time the decision was made. 


* Triage into roughiy three groups: 


Model for China visitor visas to be deployed in February 2018. 


Other predictive models to be deployed by late 2018. 
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Deployment Design: China Temporary Resident Visas 


The current processing flow for eApps looks like this: 


GEMS 


F 


| i f Reviews 
Application ig | Promoted in | application 
F Í : ened 
received by IRCE | i Decision 


i j : | recorded in 


GEMS 


Ideally, a predictive model would be integrated within GCMS as follows. This is the fully automated system that has been discussed for a 
few years, dating back to the Integrated Network Project. 
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However, we must settle for a simpler solution in the short term 


* While the previous slide illustrates the ideoi solution, integrating a madel in GCMS would take 
several years of development. it would also cost millions of dollars more than the current project. 


« The approach for deploying models in the Advanced Analytics project will be simpler and less costly 
by leveraging the copy of GCMS data stored daily in the Enterprise Data Warehouse (EDW). 


GCMS Adi operations are 
manually recorded by 
officers in GEMS 


Application | Promoted in 
| received by IRC | GEMS | a 


Contents of 
GCMS copied to 
dota warehouse 

daily 


We can’t do this ideal system because of the issue of admissibility assessments, but also 
because such a system would be very costly and take years to implement. 


EDW came up with a clever alternative for deployment which is simpler and less costly. It 
leverages the copy of GCMS data stored daily in the Data Warehouse (EDW). 


Very important to keep in mind: Because the model cannot interact with GCMS, it cannot 
make any changes to applications on its own; officer intervention will be required. 


e That also means that the results of the risk triage (e.g. colour of the bin) will not be 
stored in GCMS unless IN or CN see fit to ask officers to enter that information 
somewhere. 


e EDW will maintain a copy of the results of the risk triage, so it will always be possible 
for us to go back and review how applications were triaged and prepare reports on 
this. 


e Maybe we can make the risk triage available in Answers or Cognos so that you can 
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prepare reports. 
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Customizing the model’s output for officers 


The output from the model need not be limited to the colour of the bin. We can add other information 
to reduce clicks in case management system, For example: 


For example, for each application, we can display: 
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We can incorporate this quite easily into the report received by each office. But we don’t 
want to overload officers with too much information either. Tell us what would be useful 


and we'll make it happen. 
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ELECTRONIC TRAVEL AUTHORIZATION (PROOF OF CONCEPT) 


Proof of Concept risk assessment system in development 

¢ Predicts applicant likelihood of future adverse events. 

* Long-term working concept is to flag a small number of high-risk 
applications for additional review by human officers. 
Short-term, same system enables identification of risk factors and 


potential fraud trends. 


initial model prediction results are encouraging suggests some “low- 


hanging” high-risk cases. 


Several eTA-specific challenges: 
Limited organizational capacity to expand manual review of eTAs 
Light touch program that collects little date about mostly low-risk visitors 
Program design necessitates streaming/real-time scoring 


Extreme rare event targeting (often fewer than 1 in 1000 target cases} 
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eTA Risk Scoring and Fraud 
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Project, 


The Goal: “Eat the haystack, spit out the noedies" 
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Scoring Engine 
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Main Dataset 


Number of records: 120,114 
Dates covered: PA final decision date from 2013-01-02 to 2017-01-13 
Number of fields: 158 
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Stats on Family Class 
Applications finalized in 2013-2015 
Total applications: 142,892 
| Application Category 

Value Proportion % Count | 
EC D 9166 130978 
FCC 7.64 10913. 
FCE] 0.7 1001 
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Analytics: Opportunities for 


CIC 
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1.0 Introduction 


The term “predictive analytics” describes the process of discovering patterns in past behavior to 
predict the outcome of future events. It combines applied computer science, statistics and operations 
research to drive smarter decisions by extracting actionable insights from the vast quantities of data 
captured by government departments. At CIC, a handful of projects that enable the use of predictive 
analytics are underway; these projects have a formal project plan and schedule in place and have 
funding provided by Memorandum to Cabinet (MC) and Treasury Board submissions (see Annex 3 for 
overview of the DPAT projects and under which funding envelop they are funded). 


DPAT projects for Temporary and Permanent Resident lines of business are: 
1. Express Entry (Phase 1 & 2) 
2. ETA (Phase 1 & 2) 
3. IN Project: 
a. TR pilot (VR-Extension, International Experience Canada (IEC), Student 
b. Full TR (All TR’s; i.e. TFW/LCP, Temporary Resident Visa (TRV)and Study permit) 


Predictive analytics maximizes the Department’s resources, helps prevent fraud, and enhances 
client service. 


Predictive analytics will reduce the labour intensity of application triaging and processing as well as 
minimize the time to a decision for a significant number of cases. 


2.0 hy Predictive Analytics in OPMB? 

OPMB-PMU has a great experience in Business Intelligence (Bl); providing deep, sophisticated 
analytics on large volumes of actual and historical data. OPMB-PMU BI tools (such as Cognos10, SPSS 
and R) run advanced analytics that were previously impossible or impractical. OPMB-PMU BI 
consolidates all analytics activity to where the data resides, enabling CIC to react quickly and accurately 
on urgent requests and operational changes. Finally, OPMB-PMU has network wide experience in 
workload distribution and in providing information to all parts of the Operations Sector. This makes 


housing Predictive Analytics within OPMB strategically imperative. 


Currently, CIC makes use of Dashboards both inside and outside of GCMS Answers, in order to maintain 
workload distribution and program integrity. Other areas in CIC have developed similar tools. While 
these areas can be large parts of the network, they are not network wide, nor do they have the 
integrated historical information that OPMB has available to it. This adds strategic importance to 
housing Predictive Analytics in OPMB; it has more information on hand to not only perform network 
wide data mining, but to also perform longitudinal data mining. This is critical: the more application and 
longitudinal information missing in the data, the greater chance there is that important application 
profiles are missed. OPMB has far fewer limitations on what it can achieve with Predictive Analytics than 
any other area in the Operations sector. 
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OPMB is unbiased in the face of resource re-allocation, as it does not have any resources 5.16(2) 
physically performing operations. Bias is the bane of all information analysis. We cannot also neglect the 
fact that advanced predictive analytics users and groups that have deployed models into production are 
all adamant about the crucial need to have the predictive analytics function sitting in the business 


group, without being run and managed by the clients themselves?. 


Figure 1 ~ Data Science 


ow 


Machine ES 
Learning A ‘Dy 


Note: Hacking skills are defined as the ability to make software produce the best result (i.e. thinking “outside of the box”; 
frequent testing). 


3.0 


PAT Co 


nts and Resources 


3.1 Funding: 

Funding to develop Predictive Analytics capacity and function in CIC has been provided by 
Memorandums to Cabinet — TB submissions. Some funds are temporary (some of Express Entry for 
example), with others will be ongoing for delivering projects and establishing an ongoing predictive 
analytics function within CIC so that models can be maintained through regular monitoring and 
improvement. 


DPAT currently has slightly over $800,000? for each of the first few years to establish the team and the 
function, and the ongoing funding is expected to be approximately $700,000. Of note, though that the 
IN Project is currently expanding beyond TR, and DPAT did not receive funding for work beyond TR and a 
few PR lines. The project and ongoing funding might may change in the near future. 


+ (See Predictive Analytics Trip Report from Washington) 
* For the current projects. 
$ Of note, the USA as well as other countries, has a budget of millions for the similar teams and functionalities. 
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SA Team: 


The team was officially formed in March 2014; first with two full-time resources, now with 6.5. The team 
will soon to be expanded to 8.5 full-time resources. 


PAT Projects, as of March 2015 


AT Express Entry 


DPAT uses evidence-based indicators and 
works with other divisions such as OMC (PID, PRPD, TRPD), CPR, SPP-Admissibility, IR and SPO. 


Express Entry predictive analytics work has two phases. For Express Entry Phase 1, completed in Dec 
2014, the DPAT used proxy data to develop models and business rules. The DPAT worked closely with 
OMC (PRPD and PID) as well as with SPO, IR and CPR, to provide data and validation exercises for the 
business rules to be used at launch for triaging and assignment of Express Entry Applications. Business 
(OMC) now uses the rules that we developed in cooperation with the DPAT to process Express Entry 
cases. 


The DPAT will soon be working with CPR and Operations to use the rules to a deeper level by using 
GCMS Answers to create dashboards that will allow applications to be grouped and assigned, using the 
dashboard. Work has started in DPAT, and a first meeting was held with CPR‘. 


After the launch of Express Entry in January 2015, the DPAT is now waiting for few months to collect real 
data and validate or re-build the models based on proxy data. Once enough EE applications are 
finalized for development of predictive models, Express Entry phase-2 will start. An Express Entry 
model is expected by late Fall 2015 for implementation, and production for auto-triaging by the spring 
of 2016 (pending technical solution date and details). 


See Figure 2 (below) for illustration of what we could do when we have integration with processing 
system: 


* See the Express Entry update deck 
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Figure 2. Express Entry Future Triaging Through Predictive Analytics 


Business Rules 


4.2 Electronic Travel Authorization {eTA} 


Other divisions, such as OMC 
(PID, PRPD, TRPD), CPR, SPP-Admissibility, IR, and SPO, are involved in the process of developing the 
predictive models for eTA. 


The eTA project consists of two phases. eTA phase-1 is now finalized. We have produced and delivered 
to SPO business rules they could use at launch, based on proxy data (there is no eTA data for now, as 
eTA does not exist, so DPAT uses proxy data to build models). eTA phase-2 will be starting in October 
2015°, a few months after the implementation of the eTA (6 to 9 months, depending on if eTA starts 
with a voluntary period, as this will reduce the number of cases and will have an impact on the 
population most likely), in order to collect real data set that will be fully representative of the “real” eTA 
population. 


At this point, the implementation date is targeted for the spring of 2016. 


5 Or even later, given the CR recently submitted to postpone eTA. 
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4.3 IN Project 


4.3.1 Visitor Record-Extension 


This work is to be used to run the pilot for IN Project for TR Pilot (IEC, 
VR-Extension and Study permit). 


Delivered to SPO and Ops: 


® Predictive Analytics Models for three scenarios, based on the avallahiity of historical data, and: 


4.3.2. International Experience Class (EC) 

This project is now closed, as a draft model was delivered to business. As usual, other divisions such as 
OMC (PID, TRPD), CPR (OSC) and SPO, were involved in the process of developing the predictive models 
for IEC. 


Delivered to SPO and Ops: 


e An auto-triage model was developed. GCMS Answers replicates the model for OSC, and they will 
start using this in production for the first batch of applications for this year. 


A second phase of IEC will be starting in June 2015 in order to Improve the model and identify the best 
model for [EC by Incorporating historical data. 


4.3.3 Students 
Work on this part of the TR Pilot for IN project is close to completion. A draft model is under 
development and is due to be delivered to the clients, In July 2015. 


4.34 TRY and TFW 
Development of predictive models for TRV and TEW wil start in May and July 2015, respectively. 
Done by DPAT, for Updating Senior Management, as year-end report, P. 7 
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Figure 3- Example of TR Triage and Auto-approva! 


se Business Rules 


5.0 Engagement and working with SIMB /IT: 


This will also be used to complement risk 
and business rules. 


Requirements for Predictive Analytics processing in GCMS and in EDW have been developed by DPAT 
and are under review. Functionality for changing business rules (the models) needs to be independent 
of GCMS releases and needs to be centrally modified by the program in order to control program risk 
or in order to manage workload. Just-in-time quality assurance will also be made available. Recently, 
INP’s IEC has highlighted the need for this functionality since Canada has numerous MOUs by country 
which control the IEC program; furthermore, they are often influenced by politicians (uninfluenced by 
GCMS releases). 


OPMB has recently acquired the free crowd built software “R”f. This is the future of math and stats 
computing; R has roughly 100 million users worldwide. It can perform almost any mathematical or 
statistical function, including workload optimization. OPMB uses “R” to also replicate the SPSS Modeler 
models, for quality assurance purposes. 


6.0 Engagement of other departments and other countries: 

Predictive analytics is a fast moving technology, and maintaining external partnerships is important to 
ensure that business objectives are met and that technology focused activities are aligned with the 
broader CIC modernization vision. With this in mind, OPMB initiated and chairs the Analytics Coalition; 


6 R package is used by Australia. 
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a collaborative effort with 13 other government departments and the Privy Council’s Central 
Innovation Hub. 


The PCO Innovation Hub heard of OPMB’s work on Predictive Analytics, and in November 2014 
submitted a representative for our Analytics Coalition. The PCO will do a presentation at the next 
Analytics Coalition on an idea for “soft” peer reviewing Predictive Analytics work in the GOC. The 
Industry Canada Predictive Analytics team, which has led a recent multi-departmental DM meeting 
{including our DM) on Big Data, is also a member the Analytics Coalition. OPMB and EDW has also 
helped them to prepare for a multi-departmental ClO meeting on technical issues surrounding Big Data 
and Predictive Analytics. 


Finally, the DPAT has had dedicated meetings with to leverage what 
each organization does. The DPAT has also engaged in conversations with the UK (by CBSA LIO) and the 
Australian governments (directly), and visited the US last year. The DPAT also participates in 
government-centric Predictive Analytics conferences. 
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rtunities for Success 
Decision making processes at CIC can be enhanced by using predictive analytics models, which offer the 


following benefits: 


e Increases operational efficiency by reducing the number of paper based, manual business 
processes. 


e Decreases the effort required to do quality assurance. 


e Enables triaging and organizing application workload for optimal throughput. 
e Can be used as legal evidence to support CIC. 


Business user satistaction and an improved 
client experience are also key success factors. Long term success will be evidenced by a pervasive 
culture of analytical thinking at CIC, mathematically sound practices, and the capacity to participate with 
other government departments on predictive analytics projects. Key aspects of successful projects as it 
relates to Data and Predictive Analytics: 


Simplify the concept of predictive analytics for business users and managers. 

A consistent process for collaborating with the business stakeholders 

Specific business outcomes for predictive analytics work 

Adequate IT support 

Proper communication of the results of predictive modeling to business stakeholders 


Oy: 2 ONE 


Having timely insights from predictive models in front of the business staff that can best use 
them 

A closed-loop feedback process for improving and updating predictive models. 

8. Executive endorsement of the results and for data driven decision making 
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Annex 2: Seven Steps for Predictive Analytics Projects 


1. Collect Data and Business Requirements 


2. Conduct preliminary Data Analysis* 


3. Develop Predictive Models* 


4. Validate Predicative Models* 


5. Conduct Statistical modeling of risk/complexity 
scoring 
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Presentation Overview 


IRCC's Challenge with Processing Times 


Challenges 


The Next Steps 


Overview of the presentation; can be read line-by-line or simply communicated. 
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IRCC’s Challenge with Processing Tir 


IRCC's current processing capacity is not Projections of Temporary Resident Applications 
sufficient to address demand in a timely (Unofficial) 
manner. 


Pressures will intensify as the volume of 
applications grows. 


TR volumes could rise by I5% between 2015- 
16 and 2021.22. 


Keeping pace with rising volumes isn't 
enough; the Prime Minister has asked IRCC 
to reduce processing times. 
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What is Predictive Analytics? 


Predictive analytics consists of using computer technology to analyze past 
behaviours; through this analysis, we can discover patterns that enable 
predictions of future events. 


* Statistics + Supports evidence-based decision making 
» Computer Sciences by exploiting vast amounts of data; 


5 r + Substantiates decisions j io) 
+ Operational Expertise bstantiates decisions made by intuition or 


gut feeling; 
* Specialized Software » Enables faster business processes; 


* Business Analysis + Increases consistency in decision making: 
+ Helps maximize the use of resources; 


+ Generates synergies between business 
expertise and statistical analysis. 


Ie pee, See. Canada 
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Predictive Analytics at IRCC 


The Data and Predictive Analytics Team (DPAT) is building predictive models for many lines of business. 


DPAT is working with program experts and stakeholders to understand the decision-making process for 
each line of business, and the importance of each variable on the final decision (e.g. age, reported 
income, travel history, etc.i. 


DPAT has established strong relationships with other federal departments (including CBSA, CRA, and 
PCO} and countries (including in the U.S., the U.K. and Australia} to understand best practices. 


Canada 


Our data scientists are highly skilled (Master’s or PhD’s) on how to 
rigorously apply statistical techniques. 


Senior management and the lines of business would have to define what 
constitutes an acceptable margin of error. 
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Under the Hood 


For each line of business, our data scientists choose the 
type of model that welds the highest accuracy. 


So far, decision trees using the C5.0 algorithm have 
always been the most accurate. 


Such models generate clear decision-making rules, far : 
example: : ses @ 6 


so $ G00 9 6608 
CIEE cone 
LEE SIRS REC GENE ae 
GCS MELE CH ENN 6 
ee eens one 


Ge Sy 


+ The rules can be very granular with many conditions, 
reflecting the nuances of real-life decisions. 
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Additional Benefits of Predictive Analy : 


The criteria in these rules can also be used to identify risk 
indicators to support pragram integrity. 


We are already working with ana line of business to identify 
characteristics of high-risk individuals. 


Future uses of predictive analytics: 


~  friprove workload distribuent automaticaly assigning cases 
tothe appropriate level of officer 


These benefits are closely tied to the vision of the 
Integrated Network Project, in which predictive analytics 
plays à key role. 
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Predictive Analytics in Action 


Includes TR and PR lines of business. 
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Predictive Analytics in Action 


Assumes no improvement in forecasting accuracy of models over time, which is pessimistic. 
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allenges Faced by Predictive Analytr 


IT Architecture for Predictive Analytics 


«Need to build the IT systers that wii allow real-time information 
flaws between the predictive models and GOMIS, 


* Additional funding is required, otherwise predictive analytics cannot 
be deployed to help reduce processing times. 


Creating Support at the Program Delivery Level 


+ Officers must be ready to smoothly transition to analytics-driven 
business environment. 


* A change management strategy is being developed. 


- -o pages cece 
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Work with SIMB to define IT requirements; 


Test existing models in real time, once IT 
systems are implemented; 


Develop new models for other lines of 
business; 


Develop models for workload management; 


Continue stakeholder engagement and 
outreach; 


Support IN Project's change management 
strategy. 


Canada 


A-2018-09320-000545 


Immigration, Refugees Immigration, Réfugiés 
d Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
4 tion divulguée en vertu de la loi sur l'accès à l'information 
s 143 


Some notes on number of cases needed for testing or revision of 


For Express Entry, DPAT should be able to conduct the first testing and if necessary revision of the model 
when actual FinDec will be available for at least 3,000 applications. This was estimated by conducting 
several trials on randomly taken sample of 2% (2,936) of FSW data set (the most challenging to model) 
and the results were proven to be reasonable and valid. The monitoring of Express Entry admissions on 
weekly basis is necessary to establish the time period for required actual data be available. 


As for March 15, 2015, CIC has received 737 applications from invited candidates (about 21.5% of 
invitations). 


Taking into the consideration this recent trend in finalizing applications, DPAT believes that testing and 
revision of the model will have to wait several months; with possibility to wait to the end of 2015. 


Trials: 
The following is the presentation of results for FSW model for 2% of data that is 2,936 applications. 


There were five trials conducted to see if the model still results in accurate prediction for five randomly 
selected 2% samples of data set. 


Trial 1 
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Trial 4 
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Trials Results 


The accuracy of the model varied insignificant from 86% to 89% for five randomly selected data samples. 
This indicates the robustness of the model for 2% sample of the data. 


Next steps: 

1- Integrate new information (from current holdings) to increase accuracy. The new model will be 
built using more fields and historical information. This will increase accuracy of the model as it 
was observed from the work done for other LoBs. 

2- Re-evaluate number of applications finalized by end of June 2015 in order to plan the work and 
evaluate what will be possible this calendar year. 
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What could be emphasized and relevant 
to DPAT from PR e-apps and Express 
Entry Training Summary 


e GCMS will become the only tool to monitor and manage the work performed 
on FSW, PNP and CEC applications. 


e Application processing will become multitasking with simultaneous actions 
and not really geographically based. 


e Files will become tracked and monitored through ‘state’ rather than through 
paper file location. 


e For our ‘bins”, it would be beneficial to assign cases with the knowledge of 
this “state”. 


e It said that GCMS dashboards cannot monitor the complexity of files or the 
complexity of the work done and regular communication between supervisors 
with business knowledge and employees is required. DPAT might participate 
in these communications regarding the complexity of “cases” since the 
Integrated Network Project when we have to deal with workload distribution 
based on complexity. 


e EE eligibility will not have any details as why eligible or not eligible besides 
Status options - Obsolete, Received, or Rejected and cannot be useful for 
DPAT (page 25). 


e For FSW, any principal applicant who identifies their primary occupation as 
“Pharmacist” will be able to apply only under the Arranged Employment or 
Ph.D. stream since NOC 3131for pharmacist is not on the current list of 
eligible occupations. 
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Techniques Explored for Modelling Rare Data 


General Notes: 


1) 


2) 


3) 


4) 


5) 


6) 


7) 


8) 


1) Almost all of our business problems are binary classification problems. We may occasionally 
look at unsupervised approaches, but we generally don’t do regression. 

2) Almost all of our data is discrete/categorical. Many of the fields we use are binary. 

3) Most business problems have significant class imbalance (roughly 0.1% to 15% of the data in the 
minority class). 


C5.0 Decision Trees: 

a. Provides generally good results compared to other methods we’ve explored. 

b. Provides interpretability of the final ruleset, which is valuable from an operational 
perspective. 

c. Relatively well-suited to imbalanced data when used with a mixture of cost-weighting and 
pre-balancing of the data (over/under sampling). 

d. Cost-weighting reduces interpretability of rules. 

Other Decision Trees: 

a. Some exploration of other decision tree methods (rpart, CHAID, etc.). Test results have 

generally been inferior to C5.0. 
Classification based on Association Rules (apriori): 

a. Builtin R. 

b. Well-suited for imbalanced data as it doesn’t requiring oversampling or undersampling of 
minority/majority classes. Simple to interpret and explain to others. 

c. Computationally expensive and high possibility of overfitting given the large complement of 
rules being generated. Rule space explodes exponentially in the number of features 
examined. Requires discretization of continuous features. Less developed in the literature. 

Decision Lists (SPSS Modeler): 

a. Modeler-specific algorithm functionally very similar to the association rule technique 
mentioned above. 

b. High Interpretability. 

Random Forest 

a. Some dabbling with random forest. Has similar balancing challenges as other decision tree 
methods and is harder to interpret/explain to non-technical people. Computationally 
expensive. 

Logistic Regression: 
a. We’ve experimented with this in the past, though no work recently. 
Other: 

a. We’ve done relatively minimal exploration of a variety of other modeling techniques. We 

have not investigated these in depth and they may represent additional opportunities. 
SMOTE (data balancing): 

a. We’ve explored this method for balancing data using synthetic oversampling. The results 
have been mixed or slightly positive improvements to the models. 

b. Most of our data is categorical, which may not lend itself well to this technique. 
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RPXXX Predictive Analytics in Processing Applications 


AVersion Date Details 
V1.0 02/19/15 BPM first draft 
V2.0 03/23/15 
V3.0 03/27/15 
V4.0 04/7/2015 Incorporated MJ comments, ready for consultation 
V5.0 04/15/2015 Incorporated comments from MJ and the business 
V6.0 04/20/2015 Incorporated comments from MJ and the business (SPO and OMC) 
v7.0 04/21/2015 Consulted with SPO and EDW 


Notes on current version: 
1. The steps in this BPM do not reflect all user control rules. 


Recurring Sub-Processes referenced by this model 
RP001 Search 


RP003 Create 
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ITRIPR Triage and Auto 


Application goes through 
process as soon received 


1.0.0 


Create Application and 
Basic Name search 


12.0.8 Auto-YES (XX%) 


To be on before 
deployment. Off after. 


17.0.0 
| Silent Mode | 


See GCMS RP001 


Le] 18.0.0 

= Finalize O 

QO 200 Application 

© > Client Search for 

See GCMS RP003 Immigration 
and others History 
See GCMS RP80 
3.0.0 9.0.0 10.0.0 11.0.0 
Create Output for Apply Rules to Categorize Send for 
Predictive Analytics Dispatch cases Applications Processing 
À 
g o rt 
co = . 
Se € Solution TBD 
9J 
x 
wW 
3 
© 4.0.0 Model will create scores to 
> Receive Inputs from be sent to GCMS 
mn GCMS 
© o 
Se 
© Oo 
£ wn 
T 

v 5 y 
Š 5.0.0 6.0.0 å 7.0.0 8.0.0 “If boxes 13.0.0 and 14.0.0 is removed, this becomes a Express Entry BPM for 
5 Prepare and Validate Create Flags and PURE Send Inputs to GCMS re ; PR Triage | í 
D Data Derived Values ty {Scores and Bins) All the shaded boxes will be documented by SPO. DPAT will provide scores 
a and Bins by mode! only. 
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Hybrid Risk Assessment 


o Mitigation of known risk factors, based on historical and emerging intelligence 


o 


: Data-driven machine intelligence for risk triage and risk factor identification 
Hautes 


— 
occa 


Automatic triggers for suspicious data patterns 


ya Identifying fraud and gaming behaviour (e.g. multiple conflicting applications); 
Fabel Connecting points to broader networks of organized fraud 
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OPPORTUNITY ASSESSMENT: eTA 


ADVANIIED MAL THIS LAB IROG 


AUTOMATED 
PROCESS 


DEPLOYMENT COST 
MEDIUMHIGH 
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http://gcdocs2.ci.gc.ca/otes/cs.exe/224034787/smartDropout.PNG?func=doc.Fetch&node... 19/09/2018 


Pages 564 to / a 1008 
are withheld pursuant to sections 


sont retenues en vertu des articles 


16(2), 16(1)(b) 


of the Access to Information Act 


de la Loi sur l'accès à l'information 
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kengken Gada erigat Caridis 


May 2015 
Prepared by OPMB-PMU-DPAT 


Predictive analytics have proven value in private industry and government 


Predictive Analytics are a key piece of the modernization strategy 
DPAT is working closely with partners: SPO, PID, CPR, IR and OMC 


CIC is at the very beginning of using predictive analytics 
We are catching up on other countries (USA and Australia for example) and other 
departments ( ‘or ex.) 


Expectations need to be set carefully 


Predictive analytics can be transformative for an organization, 
but it takes a long time to put everything in place 
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Predictive Analytics ~ Overview 


What is predictive analytics 

Pronos aivalylice at CIC 

Wry is Predictive analytics uaportant for CL 
Why Predictive Analytics in Operations 
Engagomont of Key Stakeholders 

Projects update 


Example. Predictive Analytics Draft Model for YRexension 
DPAT: Next steps. 

Annex: Predictive Analytics Aporoach and Method 
Anne: Data and Predictive Anaic Roadmap 


Anneri Key steps for Success 
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What is Predictive Analytics? 


» Predictive analytics is the process of discovering patterns in historical 
data and behaviours to infer the likelihood of future events and to 
predict probable outcomes 


Drawing on a combination of... Benefits 


Computer Sciences » Formally substantiates business decisions made hy intuition ar gut 


feeling 
Statistics $ . 
Combine operational businessknowledge and expertize with 


Gperational expertise statistical analysis 


Specialized Software 


Helos maximizing the use of resources 


Drives smarter decision making By extracting insights fram tha vast 
quantity of dats available in government agencies. 


Canada 


Explains definition and how it works. Also talk about concrete benefits in any business 
doing predictive analytics (Walmart, for example, or Google, that both use that now in their 
business to save money, profile their consumers and improve resource allocations) 


The benefits of combining operations expertise with empirical data and statistical analysis 
to build a predictive model: 


* Increasing processing capacity 


* Facilitate program integrity exercises 
* Deliver optimal client service 


Forecasting vs Predictive analytics: 
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Standard reporta + Bataanalysis 


3 aien a? 
manual prosses 


and modernisation 


hue parses 


Predictive Analytics at CIC have two important goals: 
a. 


1. Enable CIC’s modernization agenda: 


Integrated Network Project, eTA, Express Entry, 
and more...; 


° By reducing the labor intensity of application triaging and 
processing as well as minimize the time to a decision for a 
significant number of cases. 


2. Help to detect and reduce fraud and contribute to program integrity 


Predictive Analytics at CIC have two important goals: 
1. 


2 
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Predictive analytics at CIC will: 
i, Enable CiC’s modernization agenda 


2. Help ta detect and reduce fraud and cantribute to program integrity 


3. Help CiCto cope with increasad volume of applications 
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Predictive Analytics in Operations 


1. Lead: OPMB- PMU 
+ Established expertise and experience in Business intelligence (BI) providing deep, sophisticated analytics on large 
volumes Gf actual and historical data 
+ Network wide experience in workload distribution in collaboration with the netwark and sector. 
+ Situated a impartial and objective enabler af operational resource allocation, 
. 


+  OPMB is the lead for Data Gavernance in the department, in partnershig with RAL 


2. Key partners and clients: 
* OMC: Program expertise {TRPE PRED and programs integrity (FIDY 
¥ SPA: Project leads ancl clients 
CPR including regions when seeded) and IR ertain processing and subject matter experts fo 
gathering 
SIME: support iT systems and data 
Other departments and jurisdictions: for lessons learned, best practices and information sharing. 
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Our stakeholder engagement” fs three told 
1. Bi-weekly working group meetings with partners: 
» OPMB/ DPAT identifies and meets regularly with partners and stakeholdars ta collect business 
requiraments for building predictive modals that ara aligned with CIC business and with 
madernizationagends. 


Working groups are by line of business and by projects (IN orciact ~ JEC, Extension, Students, eTA, 
and Express Entry} 


2. Stakeholder engagement sessions 
» The purpose of early engagementis to build relationships, explain what Predictive Analytics is and its 


benefits; and te establish protocols far delivery and communication between the teams. 


3. Engagement and sharing best practices with other departments and other countries; 


o We Rave had meetings with many groups of CBSA; we chair a quarterly Date Analytics Coalition 
regrouping 14 government denartments and we have engaged with foreian governments such as the 


“our stakeholder engagement started April 2014, 


stakeholder engagement* is three folded: 

1.Bi-weekly working group meetings with partners 

_ Every DPAT project (models) starts with a clear definition of business goals 
and a partnership is formed with the line of business 

_DPAT Discuss findings with the business on regular basis for validation and 
to combine qualitative business knowledge with empirical statistical research 
2. Stakeholder engagement sessions 

-bi-monthly stakeholder engagement sessions to discuss findings with the 
business on regular basis for validation and to combine qualitative business 
knowledge with empirical statistical research 

-Program experts, policy and administrative teams need to be involved to 
ensure successful project delivery 

3. Engagement and sharing best practices with other departments and 
other countries; 

-Held numerous meeting with CBSA 

-Initiated a collaborative effort with other government departments and 
chairs a Data Analytics Coalition with key government departments and 
agencies 
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Projects Update 


À: Express Entry 

DPAT analyzed applications processing to identify the most important indicators and build a madel with the goal of 
allowing automated triaging and distribution af applications. 

* Express Entry-phase 1: Preliminary model done and business rules extracted on time for launch. 
Express Entry-phase 2: will startin g few months, when more Express Erry applications have been processed 
and finalised, 

& The goal is to have a solid model thar we can deploy inte productianfor auto-triaging of applications. 
{tentative date for go-live is Suenrear 2025} 


2- Electronic Travel Authorization 


Speaker: Marie-Josee 


and to allow for groupings to be done for automated 
distribution of applications. 


1- Express Entry: 


1- Express Entry-phase 1: COMPLETED 
*DPAT developed models for FSW, CEC and PN and delivered 


Business rules for launch of Express Entry 


*DPAT is working with CPR and Ops to use the business rules a 
level deeper by using GCMS Answers to create dashboards that 
will allow applications to be grouped and assigned, using the 
dashboard. 


2- Express Entry-phase 2: will start in a few months 
*DPAT is now waiting for few months to collect real data and 


validate or re-built the models developed based on proxy data. 
Once the data is available, Express Entry phase-2 will start in June 
2015. 
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Express Entry auto-triaging model will be deployed in spring 2016. 


2. eTA phase-2: will be starting in Oct 2015 (or even later now, 
with the new date) 
- Once we have phase 2 data; i-e real data coming from eTA 
applications. 


1.TR Pilot: for deployment in the Spring/ June 2016. 
1.1. ICE 
IEC Working Groups held with stakeholders and draft model and GCMS 
Answers case-statements was delivered to OSC. Currently participating in 
INP IEC Phase 1 implementation (the digitization of IEC). 
DPAT will rebuild the model just prior to launching the Phase 2 pilot in 
2016. 
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1.2. VR-Extension 


VR-extension project started in Dec 2014 and is finalized in February 
2015. 

DPAT delivered Predictive Analytics Model to SPO and OPS. 

1.3. STUDENT 

_DPAT is starting the working groups with stakeholders to develop 
predictive model for auto-triaging 


2.Full TR 


TRV_and TFW (including LCP) 
-DPAT will start the working groups with stakeholders to develop 


predictive models in Spring 2015 
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Projects Update (con't) | s.16(2) 
3- IN Project 


1- TR Pilot: for deployment in June 2016. 
+ IEC: Draft model COMPLETED 
+ VR-EXTENSION : Draft model COMPLETED 
» STUDENT : started in February 2015 
3 Draft model under developeri 
The models developed during 2014 and 2015 for the TR Pilot wili be retested and validated with new applications 
in the months before launch, 


2- Full TR: for deployment in January 2017 
+ The “full TRE includes : 
oo Visitors (IRV), Femupsrary Foreign Workers {IHW}, including Live-in Caregiver (LSS) 


+ TRY and TEW (including LCP): Development of predictive models will start in Summer 2015 


1.TR Pilot: for deployment in the Spring/ June 2016. 
1.1. ICE 
IEC Working Groups held with stakeholders and draft model and GCMS 
Answers case-statements was delivered to OSC. Currently participating 
in INP IEC Phase 1 implementation (the digitization of IEC). 
DPAT will rebuild the model just prior to launching the Phase 2 pilot in 
2016. 
1.2. VR-Extension 


VR-extension project started in Dec 2014 and is finalized in February 
2015. 

DPAT delivered Predictive Analytics Model to SPO and OPS. 

1.3. STUDENT 

-DPAT is starting the working groups with stakeholders to develop 
predictive model for auto-triaging 
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2.Full TR 


TRV_and TFW (including LCP) 
_DPAT will start the working groups with stakeholders to develop 


predictive models in Spring 2015 
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EXAMPLE : Predictive Analytics draft model for VR-« 
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DPAT - Next Steps 


Deliver on Projects 
al IN Project 
- TR Pilot 
+ FULL TR 
b} Express Entry -Phase 2 
ci eTA -phase 2 


Monitor and improve models {including QA and QC of performance of models} 


Continue Stakeholder engagement and outreach 


Consult and agree on model governance 


With work IT to develop solutions 


- -o e 
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Annex 1: Predictive Analytics App “0 


Colect Gute and Businesss Beguirements 
Conduct praleninary Data Analysis 
Gevalon Predictive Models 
Validate Predicative Models. 


Conduct Statistical modeling of dak/complenity scoring 


OF NOTE : THIS IS AN ITIRATIVE PROCESS, where we can go through steps one time, but we can also go between steps and come back if 
needed. 


The Predictive Analytics approach and methods is based on the CRISP-DM phases. 


CRISP-DM breaks the process of data mining into six major phases (and we have developed and agreed on a 7 step process) 


1- Business Understanding : This initial phase focuses on understanding the project objectives and requirements from a business 
perspective, and then converting this knowledge into a data mining problem definition, and a preliminary plan designed to achieve the 
objectives. 


2- Data Understanding: The data understanding phase starts with an initial data collection and proceeds with activities in order to get 
familiar with the data, to identify data quality problems, to discover first insights into the data, or to detect interesting subsets to form 
hypotheses for hidden information. 


3-Data Preparation: The data preparation phase covers all activities to construct the final dataset (data that will be fed into the 
modeling tool(s)) from the initial raw data. Data preparation tasks are likely to be performed multiple times, and not in any prescribed 
order. Tasks include table, record, and attribute selection as well as transformation and cleaning of data for modeling tools. 


4- Modeling: In this phase, various modeling techniques are selected and applied, and their parameters are calibrated to optimal values. 
Typically, there are several techniques for the same data mining problem type. Some techniques have specific requirements on the form 
of data. Therefore, stepping back to the data preparation phase is often needed. 


5- Evaluation: At this stage in the project you have built a model (or models) that appears to have high quality, from a data analysis 
perspective. Before proceeding to final deployment of the model, it is important to more thoroughly evaluate the model, and review the 
steps executed to construct the model, to be certain it properly achieves the business objectives. A key objective is to determine if there 
is some important business issue that has not been sufficiently considered. At the end of this phase, a decision on the use of the data 
mining results should be reached. 


6-Deployment: Creation of the model is generally not the end of the project. Even if the purpose of the model is to increase knowledge 
of the data, the knowledge gained will need to be organized and presented in a way that the customer can use it. Depending on the 
requirements, the deployment phase can be as simple as generating a report or as complex as implementing a repeatable data scoring 
(e.g. segment allocation) or data mining process. In many cases it will be the customer, not the data analyst, who will carry out the 
deployment steps. Even if the analyst deploys the model it is important for the customer to understand up front the actions which will 
need to be carried out in order to actually make use of the created models. 
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Key Steps to Success with Predictive Analytics 


Simplify the concept of predictive enalytics for business users and managers. 
Executive sponsorship for predictive analytics projects 
A consistent process for collaborating with the business stakeholders 
usiness outcomes for predictive analytics work 
Adequate fT support 
Proper communication of the results of predictive modeling to business stakeholders 
Having timely insights from predictive models in front of the business staff that can best use them 
À closed-loop feedback process for improving and updating predictive models. 
Executive endorsement of the results and for data driven decision making 


A center of excellence for date and predictive analytics 
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Predictive analytics have proven value in private industry and government 


Predictive Analytics are a key piece of the modernization strategy 
DPAT is working closely with partners: SPO, PID, CPR, IR and OMC 


CIC is at the very beginning of using predictive analytics 
We are catching up on other countries (USA and Australia for example) and other 
departments 


Expectations need to be set carefully 


Predictive analytics can be transformative for an organization, 
but it takes a long time to put everything in place 
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Predictive Analytics ~ Overview 


Wher is Predictive Anslytion? 

Why ls predictive analytics inportant? 

Prechctne Analytics af OC 

Engagement of Key Stakeholders 

Project Updates 

famiglia: Predictive Makita Dealt Mode for Ventes 
À New Realty Two Wowie af BY 

DPAT Hert Seps 


Apical: Predictive Aro Ues Apres 


Speaker: Stephanie Kirkland 
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What is Predictive Analytics? 


» Predictive analytics is the process of discovering patterns in historical 
data and behaviours to infer the likelihood of future events and to 
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predict probable outcomes 


Drawing on a combination of... 
Computer Seances ‘ 
Stasis 


Cwerational exeertisea 


Specialized Software 


Benefits 


Formally substantiates business decisions made by Intuition ar gut 


fesing 
Cambine aperationa! business knowledge and expertise with 
statistical analysis 


Heips maxicing the use of resources 
Drives smarter decision making by extracting insights fram the vast 


quantty of dats available in government agencies. 


Explains definition and how it works. Also talk about concrete benefits in any business 
doing predictive analytics (Walmart, for example, or Google, that both use that now in their 
business to save money, profile their consumers and improve resource allocations) 


The benefits of combining operations expertise with empirical data and statistical analysis 


to build a predictive model: 


$ 


* Increasing processing capacity 


$ 


* Facilitate program integrity exercises 


+ Deliver optimal client service 
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Predictive analytics offers the opportunity to use data to drive decision: 


With the help of Predictive Analytics, decision making graces will ie: 


Serena Ana: 
Machine Learning 


panned Rae te “eee i 
Date Gumes “ 


ee ets p: 
“Rej 
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Standard reports Gate analysis Predictive 


Advhonraports + Damis 


A combination of sapturad, ut 


manual penemsaes 
and modernisation 
initiatives 


leweraged tor Risk triterta 

business dentié 

gurposst managed 
validated 


Predictive Analytics at CIC have two important goals: 
a. 
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Trends, ches semen, ai 
POC ae ented wane: 
she Seared date 
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Liane Enable CCS muier stration agamia 
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| 


néon date 
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SF épars Paacnae: 
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DPAT - OPMB 


Canada 
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Engagernent of Key Stakeholders 


* Our stakeholder engagement® fs three told 
1. Bi-weekly working group meetings with partners: 
» OPMB/ DPAT identifies and meets regularly with partners and stakeholdars ta collect business 
requiraments for building predictive modals that ara aligned with CIC business and with 
madernizationagends. 


Working groups are by line of business and by projects (IN orciact ~ JEC, Extension, Students, eTA, 
and Express Entry} 


2. Stakeholder engagement sessions 
u The purpose of early engagermentis to build relationships, explain what Predictive Analytics is and its 
benefits; and te establish protocols far delivery and communication between the teams. 


3. Engagement and sharing best practices with other departments and other countries; 


We have had meetings with many groups of CBSA; we chair a quarterly Date Analytics Coalition 
regrouping 14 government departments and we have engaged with foreign governments such as the 
to share techniques and best practices. 


“our stakeholder engagement started April 2014. 


Canada 


> Our stakeholder engagement* is three folded: 


1.Bi-weekly working group meetings with partners 

- Every DPAT project (models) starts with a clear definition of business goals and a 
partnership is formed with the line of business 

-DPAT Discuss findings with the business on regular basis for validation and to 
combine qualitative business knowledge with empirical statistical research 

2. Stakeholder engagement sessions 

-bi-monthly stakeholder engagement sessions to discuss findings with the business 
on regular basis for validation and to combine qualitative business knowledge with 
empirical statistical research 

-Program experts, policy and administrative teams need to be involved to ensure 
successful project delivery 

3. Engagement and sharing best practices with other departments and 
other countries; 

-Held numerous meeting with CBSA 

-Initiated a collaborative effort with other government departments and chairs a 
Data Analytics Coalition with key government departments and agencies 
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Project Updates 


1- Express Entry 
DPAT analysed applications processing ta identify the most important indicators are bel a model with the goal of 
allowing automated distribution of applications. 
* Express Entry-phase 1: Preliminary model done and business rules extracted, 
* Express Entry-phase 2 will startins few months iwher mare Express Entry applications have bees 
procosserd and finalized, TROJ 
a The goal is to have a solid model that we can deploy inte production tor auta-triaging of applications 
{tentative date for go-live is lune 2016} 
2- Electronic Travel Authorization 


+ «TA expansion: Draft modelis under development, Testing date ta be determined. Suggested GoLive is lanuary 
et? 


ie oe, eo 


and to allow for groupings to be done for automated distribution 
of applications. 


1- Express Entry: 


1- Express Entry-phase 1: COMPLETED 
*DPAT developed models for FSW, CEC and PN and delivered Business 


rules for launch of Express Entry 


*DPAT is working with CPR and Ops to use the business rules a level 
deeper by using GCMS Answers to create dashboards that will allow 
applications to be grouped and assigned, using the dashboard. 


2- Express Entry-phase 2: will start in a few months 
*DPAT is now waiting for few months to collect real data and validate 


or re-built the models developed based on proxy data. Once the data 
is available, Express Entry phase-2 will start in June 2015. 
*Express Entry auto-triaging model will be deployed in spring 2016. 
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2. eTA phase-2: will be starting in Oct 2015 (or even later now, 
with the new date) 
- Once we have phase 2 data; i-e real data coming from eTA 
applications. 


3- 


1.TR Pilot: for deployment in the Spring/ June 2016. 
1.1. ICE 
IEC Working Groups held with stakeholders and draft model and GCMS 
Answers case-statements was delivered to OSC. Currently participating in INP 
IEC Phase 1 implementation (the digitization of IEC). 
DPAT will rebuild the model just prior to launching the Phase 2 pilot in 2016. 
1.2. VR-Extension 


VR-extension project started in Dec 2014 and is finalized in February 2015. 
DPAT delivered Predictive Analytics Model to SPO and OPS. 
1.3. STUDENT 
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_DPAT is starting the working groups with stakeholders to develop predictive 
model for auto-triaging 


2.Full TR 


TRV_and TFW (including LCP) 
-DPAT will start the working groups with stakeholders to develop predictive 


models in Spring 2015 
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Project Updates (con't) 


3- IN Project 


1- TR Pilot: for deployment in June 2016. 
+ IEC: Draft model COMPLETED 
* VR-EXTENSION : Draft model COMPLETED 
» STUDENT : Draft model Completed. 
the models developed during 2014 and 2015 far the TR Pilot will be retested and validated with new applications in 
the months before launch, Testing date will be in February 2016 and the suggested Go-live date is June 2046, 


2- Fall TR: for deployment in January 2017 
* The “full TR” includes : 
& Visitors {TRY}, Temporary Foreign Workers (TEW , iecluding Live-in Caregiver GC 
* TRV and TFW fincluding LES) Draft model s are under develmpment, Testing date will be March 2016 and the 
suggeetn GoLive Cate ip January AT. 


ie oe, eo 


1.TR Pilot: for deployment in the Spring/ June 2016. 
1.1. ICE 
IEC Working Groups held with stakeholders and draft model and GCMS 
Answers case-statements was delivered to OSC. Currently participating in INP 
IEC Phase 1 implementation (the digitization of IEC). 
DPAT will rebuild the model just prior to launching the Phase 2 pilot in 2016. 
1.2. VR-Extension 


VR-extension project started in Dec 2014 and is finalized in February 2015. 
DPAT delivered Predictive Analytics Model to SPO and OPS. 

1.3. STUDENT 

-DPAT is starting the working groups with stakeholders to develop predictive 
model for auto-triaging 


2.Full TR 


TRV and TFW (including LCP) 
-DPAT will start the working groups with stakeholders to develop predictive 


A-2018-09320-001036 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


models in Spring 2015 
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ne Data 
Functions architecture 


Erker 


i. KONE Cane pages cee 


Historically, BI leaders have lived in a world where they centrally own and manage the data 
warehouse, and are responsible for delivering reports to the business. Today there are two 
distinct worlds, serving different types of users who have different analytical needs. One 
world provides casual users with reports and dashboards built to answer predefined 
questions using IT-certified data. The other gives power users unfettered access to any data 
to answer unanticipated questions using ad-hoc query and analysis tools. (Wayne Eckerson) 
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Two Worlds of BI (cont’d) 


Pros: 
“Data Consistency 
“Business Alignment 


Cons: 

“Hard to budd 
“Hard to change 
"Expensive 
“Politically charged 


Pros: 

“Quick to build 

Easy to change 
“Low cost 

«Politically uncharged 


Cons: 
‘Data Consistency 
“Business Alignment 


Top down. The “top down BI”, is driven by the information technology (IT) to meet the 
needs of casual users who primarily (but not exclusively) want to monitor known processes 
using dashboards that contain certified data gleaned from the data warehouse. The 
downside of top-down BI is that generating certified data takes time, money and political 
willpower, and it’s not optimized to support self-service BI. (Wayne Eckerson) 


Bottom up. It is driven by business unit heads to meet the needs of power users who are 
tasked to find new insights (not monitor existing processes) and answer unanticipated 


questions using any data, certified or not. It is agile and supports self service to a fault. 
(Wayne Eckerson) 
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DPAT - Next Steps 


Deliver on Projects 
al IN Project 
- TR Pilot 
+ FULL TR 
b} Express Entry -Phase 2 
ci eTA -phase 2 


Monitor and improve models {including QA and QC of performance of models} 


Continue Stakeholder engagement and outreach 


Consult and agree on model governance 


Develop IT solutions 


Speaker: Stephanie 
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Annex 1: Predictive Analytics Approa 
Colect Gute and Business Beguirements 
Conduct proberinary Bata Araiysis 
Dates Predictive Modes 
Wadidate Predication Models 
Conduct Statistical modeling of risk/complentty scoring 
Speaker : SK or MJ 
OF NOTE : THIS IS AN ITIRATIVE PROCESS, where we can go through steps one time, but we can also go between steps and come back if needed. 
The Predictive Analytics approach and methods is based on the CRISP-DM phases. 
CRISP-DM breaks the process of data mining into six major phases (and we have developed and agreed on a 7 step process) 
1- Business Understanding : This initial phase focuses on understanding the project objectives and requirements from a business perspective, and then 
converting this knowledge into a data mining problem definition, and a preliminary plan designed to achieve the objectives. 
2- Data Understanding: The data understanding phase starts with an initial data collection and proceeds with activities in order to get familiar with the 
data, to identify data quality problems, to discover first insights into the data, or to detect interesting subsets to form hypotheses for hidden information. 
3-Data Preparation: The data preparation phase covers all activities to construct the final dataset (data that will be fed into the modeling tool(s)) from the 
initial raw data. Data preparation tasks are likely to be performed multiple times, and not in any prescribed order. Tasks include table, record, and attribute 
selection as well as transformation and cleaning of data for modeling tools. 
4- Modeling: In this phase, various modeling techniques are selected and applied, and their parameters are calibrated to optimal values. Typically, there are 
several techniques for the same data mining problem type. Some techniques have specific requirements on the form of data. Therefore, stepping back to 
the data preparation phase is often needed. 
5- Evaluation: At this stage in the project you have built a model (or models) that appears to have high quality, from a data analysis perspective. Before 
proceeding to final deployment of the model, it is important to more thoroughly evaluate the model, and review the steps executed to construct the 
model, to be certain it properly achieves the business objectives. A key objective is to determine if there is some important business issue that has not been 
sufficiently considered. At the end of this phase, a decision on the use of the data mining results should be reached. 
6-Deployment: Creation of the model is generally not the end of the project. Even if the purpose of the model is to increase knowledge of the data, the 
knowledge gained will need to be organized and presented in a way that the customer can use it. Depending on the requirements, the deployment phase 
can be as simple as generating a report or as complex as implementing a repeatable data scoring (e.g. segment allocation} or data mining process. In many 
cases it will be the customer, not the data analyst, who will carry out the deployment steps. Even if the analyst deploys the model it is important for the 
customer to understand up front the actions which will need to be carried out in order to actually make use of the created models. 
1 À 
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Applying Machine Learning to Federal Immigration Disputes: 
A Case Study of Applications Before Immigration, Refugee, and Citizenship Canada 
(IRCC) 
Albert Yoon, JD-PhD' 
Blue J Legal 
18 June 2017 


Prepared for Department of Justice Canada 


Summary 


" Albert Yoon is a co-founder of Blue J Legal, Inc. He is also Professor and Chair in Law and 
Economics at the University of Toronto Faculty of Law. His academic research empirically 
examines judicial behaviour, the legal profession, and labour markets. He received his BA from Yale 
University, and his JD and PhD (political science) from Stanford University. Please send any 
comments to albert@bluejlegal.com. 
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Chain of Trust: Collaborative Risk Assessment for Air Traveller Programs 


Lead : Immigration, Refugees and Citizenship Canada 


Partnership: Canada Border Services Agency, 
WorldReach Software, Entrust Datacard, 
Face4 Systems, University of Ottawa 


Start-End: November 2016 to March 2018 


Funds: 


Risk 


$1,296,523 $1,496,000 $0 $2,792,523 
Project Summary: Outcomes: 
Objectives: Deliverables: 


+ A prototype demonstrated in a representative environment. 

¢ Four technology refinement reports documenting progress 
through iterative stages of prototype development 

¢ An Evaluation Report, a Technology Dissemination Plan and 
an outline of a potential pilot 


+ Demonstrate how new technologies and data 
sources can enhance the risk assessment of air 
travellers as they progress through the travel 
continuum. 

Impacts: 

Aligned to Challenge #14 

¢ Enhanced cross-agency collaboration and information 
sharing for risk assessment 

¢ Improved service experience for low-risk travellers 

+ Enhanced targeting of persons-of-interest 

¢ Improved capability for remote authentication of air 
travellers before departure to Canada 


~ 4 
Caan ram The CSSP is led by Defence Research and Development Canada's Centre for Security Science, in partnership with Public Safety Canada. Canada og 
A-2018-09320-001066 


¢ Demonstrate that low-risk travellers would 
benefit from faster service with minimal human 
contact, while high-risk travellers would be 
flagged for greater scrutiny. 
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March 2015 
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Decision tree for CEC s.16(2) 


Why CEC Dashboard 


ps 


While we don’t have the functionality to deploy a real predictive model in 
production, CEC dashboard was prepared by DMT and DPAT to test business rules 
provided by the model for possible auto-triaging and auto-assignment. 


The model has rule sets presented as If ... Then... statements that can be imposed 
on upcoming applications for triaging. 


This was coded description of the splits found by C5 algorithm (see Express Entry 
master modeling document for details). 


The outcome of application of these rules was assessed vs. actual final 
assessment of application in GCMS for accuracy in order to see if these rules could 
be used for future auto-triaging and auto-assignment. 
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E 5 
SPSS Decision Tree for CEC 


e The interactive decision tree was built for CEC similar to FSW (see master 
document for details). 
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GCMS Data Overview to Define Business 


e Number of application finalized between July 16, 2011 and August 15, 2014 
e Total number of valid records consist of 26,477 applications 


e Total number of custom fields (columns) is 2 
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GCMS Methodology - Triage 


e CEC decision tree rules as identified by the SPSS modeller were transferred to 
GCMS in a Series of case statements. 


e Currently, this method is being used to validate the accuracy of SPSS triage rules 
vs. actual final assessment of GCMS application. 
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e Example of GCMS Case Statement using SPSS decision tree rules: 
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GCMS Triage Rule Outcome 


The report outcome displays the Application Number along with: 
e Triage Rules — defined by SPSS rules 
e Final Assessment — actual final assessment of GCMS application 


Thie ig g sé 


Table id YP x 
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Note: We should not perform triaging using only business rules. We must use 
predictive models for triaging criteria. If we don’t have access to predictive 
modeling, then we may use business rules as a last resort. 
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Next Steps 


e Develop similar dashboard for FSW and PN if client need them 


e Identify feasible and optimal number of cases to test and if necessary revise the 
model once actual data available 
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Project #: CSSP-2016-CP-2276 | UNCLASSIFIED 
E = 
Unclassified 


Canadian Safety and Security 
Program (CSSP) 


Project Charter 


Chain of Trust: Collaborative Risk 
Assessment for Air Traveller Programs 
CSSP-2016-CP-2276 


Version: 1.0 


Version 1.0: 26 JAN 2017 
Page 1 of 17 
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Project # CSSP-2016-CP-2276 UNCLASSIFIED 


RECORD OF AMENDMENTS 


Record major changes between final Charter versions; commencing at Version 1, initial Charter. 


Charter 
| Version No. 


Amendment / Section Amended Entered By Amendment Date 


Version | Initial Cha 


| Steven 


26 January 2017 


Charter Revision: Project changes that impact scope, budget or schedule require a full Charter revision and sign off by all 
participants. Charter Amendment: Project changes that are minor and do not impact scope, budget (with no impact to FY 
Cashflow} or schedule require a Charter Amendment and sign off by the Lead Government Department and DRDC CSS. 
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Project #: CSSP-2016-CP-2276 UNCLASSIFIED 


INTRODUCTION 


1. Purpose 


The Project Charter is established between Defence Research and Development Canada, Centre for Security Science 
(DRDC CSS) and Project participants to acknowledge the existence of a project. The Charter outlines the parameters 
and structure of the project and includes project objectives; outcomes; impact; risks; schedule; resources; budget; and 
project organization. The Charter, itself, is not a contract. A contract or Memorandum of Agreement (MOA) must be 
issued through Public Works and Government Services Canada (PWGSC) to non-federal government partners prior to 
the commencement of any project activities. In case of conflict, any resultant contract/(MOA takes precedence over the 
Charter. 


2. Summary 


This Project Charter hereby establishes Chain of Trust: Collaborative Risk Assessment for Air Traveller Programs (CoT) 
as a Project in accordance with the Canadian Safety and Security Program (CSSP) managed through DRDC CSS. This 
Call for Proposals project is being conducted primarily for the Border and Transportation Security Portfolio. 


3. CSSP Outcomes 
This project contributes to the following CSSP Outcomes: 


e Secure/Open Borders 


e Effective Technology Assessment/Transition 


4. Background 


The risk assessment of travellers entering Canada is a growing challenge given the rise in global security threats and the 
increasing sophistication of bad actors. In particular, visa-exempt extremists were identified as a top threat in Canada's 
2014 Terrorist Report. 


This project picks-up where FASTER-PrivBio left off by taking the proof of concept application and demonstrating it in a 
prototyping environment which includes operationally relevant systems and users. 

The project addresses Challenge 14 “to enhance targeting solutions that make use of all available data, including 
electronic traveller data and biometrics, for the purposes of identifying low-risk travellers”. It focuses on international 
travellers coming to Canada using the Electronic Travel Program (eTA) and returning Canadians (both via air). The 
project contains four key areas of activity: 


e Activity 1: Augmented App and Traveller facing systems 

The Smartphone App and Traveller facing systems allow the collection of electronic data throughout the trip. 

a) Smartphone App: the FASTER PrivBio app currently allows the ability to obtain an eTA. The app will be 
extended to allow the submission of the E311 Declaration Form and will potentially support a broader range of 
phones. 

b) Kiosks and other traveller facing transactions: Check-in, boarding pass, boarding kiosk, declaration and. 
primary kiosks are used in conjunction with the smartphone App throughout the trip. Examples of these 
systems within the airports of departure and arrival will be developed. 


e Activity 2: Back-end Systems 


Backend systems allow transaction data to be stored, verified, consolidated, distributed and shared. The CoT 
system will mock-up various backend systems to demonstrate this ability. 


Version 1.0: 26 JAN 2017 
Page 3 of 17 
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Project #: CSSP-2016-CP-2276 UNCLASSIFIED 


FASTER eTA approval: The FASTER server system developed in FASTER PrivBio will be extended. 


e Activity 3: Collaborative Analytics 


+ Activity 4: Demonstration Prototype 


A demonstration of how the prototype will result in faster service throughout the continuum for low-risk travellers, 
in exchange for quality information about themselves. The use of streaming and automated border 
controls/technology will lead to more efficient processing at the various stages of the continuum, with minimal 
human contact. 


The ConOps for this prototype enables the Air Border of the Future 2020 operational vision prepared by the CBSA and 
IRCC in 2014/15 with input from the air industry. The federal government stakeholders in the initial FASTER-PrivBio 
project will continue to support this new prototype proposal with their in-kind support and direct involvement across 
research, policy and operations. 


This project offers significant potential to transform the travel experience of millions of travellers by implementing the 
ConOps to be demonstrated in the prototype in the air travel continuum. Surveys clearly indicate that consumers are 
eager to access government service through mobile or self-service models. 


Pull demand from partners could include: 


e [RCC and CBSA: enhance the security and efficiency of immigration and border enforcement programs through 
new sources of information, and collaborative risk assessment. 
e Stakeholders represented through IRCC/CBSA partners: 
o Public Safety Canada, including intelligence and enforcement bodies (Canadian Security Intelligence Service 
(CSIS) and the Royal Canadian Mounted Police (RCMP) 
o Travel/Air Industry Groups: Chain of Trust generates efficiencies by streamlining low risk travellers at airports. 
o U.S. Customs and Border Protection: Chain of Trust is consistent with the spirit of the Beyond the Border 


Action Plan. 

o Five Nations: Canada could provide an innovative solution by pushing out the border and sharing best 
practices. 

o international organizations, including the UN Security Council and International Civil Aviation Organization 
{ICAO). 
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5. Objective(s) 


The objective of this prototype exercise is to identify the necessary requirements and target the overall proposed 
technology up to TRL 7. A demonstration of a prototype will be performed in a representative environment rather than an 
operational environment. 


The CoT project will address CSSP Challenge 14 to enhance targeting solutions that make use of all available data, 

| at i. for the purposes of identifying low-risk travellers. This will be achieved 
by demonstrating a unique end-to-end technical solution that could be used to analyze the risk associated with a traveller 
from the moment an eTA application is received by IRCC to the moment when the traveller reaches a Canadian POE. 
This will facilitate the clearance of low-risk travellers, while helping to detect and deter potential terrorists before migration 
approval or entry into Canada. 


From an internal stakeholder perspective, CoT could support: 


a) IRCC admissibility decisions by using predictive analytics to analyze relevant information about travellers prior to the 
issuance of an eTA. 

b) CBSA in its mandate to enable effective and efficient screening and more streamlined clearance of low-risk travellers 
across borders. i 

c) The Beyond the Border Action Plan by supporting earlier identification of security threats. 


6. Impact on the CSSP Outcome(s) 
Secure/Open Borders 


The project demonstrates the possibility to improve border security by consolidating available information and introducing 
automated risk screening based on collaborative predictive analytics. This will help address a top national security threat 
identified in the 2014 Terrorist Threat to Canada Report. 


The project enhances border openness because risk screening will allow IRCC and CBSA to streamline and facilitate the 
passage of low-risk travellers. Moreover, the facilitation of travel through the use of a mobile phone and secure biometrics 
authentication will make travel easier. The potential benefits are significant given that eTA clients represent 3.5 million 
travellers per year and returning Canadians represent an additional 17 million travellers per year. The mobile app will also 
allow travellers to complete the E311 Declaration Form on their phone, thus further accelerating border clearance. 


Effective Technology Assessment/Transition 

The project wili allow the partners to assess how various technologies can contribute to risk triaging of travellers, including 
biometric authentication, data consolidation tools and predictive analytics. It will allow partners to evaluate how the level of 
risk evolves as the traveller generates new information while transitioning through various zones of the continuum. 


These advancements will position Canada as a leader in pre-screening immigration applicants and facilitating low-risk 
traveller movement. They will also position the partners to potentially undertake a pilot. 
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PROJECT SCOPE 


The project brings together IRCC, CBSA and industry innovators in a thought-leading approach to apply new 
knowledge/technologies for collaborative risk assessment and establishment of the traveller identity throughout the 
traveller continuum. 


7. included Work 


3) Extended ePassport testing 


4) Token Usage 


Token creation and verification using the ePassport facial image was done in the initial FASTER-PrivBio project 
for verification at the border. 


5) Faces on the Move integration 


6) E311 Declaration 


Prototyping of the FASTER mobile app integration with the E311 mobile Declaration Form will be done to illustrate 
the interoperability and facilitation of low risk traveller border entry. 


7) Cloud-based demonstration 


FASTER-PrivBio uses commercial Cloud and WAN simulation tools for testing and simulating Internet 
connectivity used by eTA travellers to connect with IRCC. Access to a commercial cloud could be used for this 
prototype to simulate access to the IRCC /CBSA web sites and FASTER connections to the central servers. 


8) Interaction of transient and permanent data record 


The collaborative risk aspect of the project proposes to establish a server component (possibly within the 
commercial cloud) to maintain a transient and dynamic record for each traveller that directly parallels the travel 
continuum. It is anticipated that this transient record, hereafter referred to as a Traveller Risk Profile, might be 
created at pre-departure and will be informed with additional information collected during the travel continuum, as 
well as with the collaborative predictive analytics risk assessments and potentially other data sources. We will 
also investigate how the status of the Traveller Risk Profile, and potential alerts resulting from the various 
interactions, might be made available to various operational stakeholders e.g. CBSA field officers (via web or 
dedicated User Interface (Ul) tools) to provide real-time indication of progress against configured risk 
benchmarks, that might imply an alteration in the continuum flow. Upon completion of processing of the traveller 
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through all zones of border control, it is expected that the Traveller Risk Profile structure would be recorded in a 
permanent Universal Travel Record (UTR) for that traveller. 


9) Passport Renewals 


Some prototyping will be explored of the online simplified renewal process for Canadian ePassports. This will 
involve scanning of the biopage in the ePassport as done in FASTER to obtain and validate the existing passport 
information as well as populate an application form. 


10) Collaborative Analytics 


The collaborative predictive analytics involves IRCC and CBSA each developing their own predictive models to 
assess the risk level of each traveller. IRCC's risk assessments will be shared with CBSA and incorporated into 
their own risk assessment. Additionally, CBSA’s risk assessments will be updated as each traveller advances to a 
later zone in the continuum, thus generating new information that could influence the risk assessment. Pending 
the results of the analysis, IRCC may be in a position to begin modifying Global Case Management System 
(GCMS) to deploy predictive analytics models for risk assessment. For Canadians returning to Canada, CBSA will 
conduct its risk assessments alone, as IRCC plays no role in the screening. 


11) Demonstration 


Facilities will be obtained to host the prototype demonstration exercises. The project team will simulate the zones 
of the air travel continuum to assess how the prototype improves the overall travel experience of low-risk 
travellers in exchange for quality information about themselves. Equipment used for these simulations will include: 
kiosk technology, mobile devices, surveillance systems, data simulators, backend servers, and ‘front counter’ 
workstations. 


8. Excluded Work 


The demonstration will take place in a representative environment, not an operational environment. 


No real traveller information will be exchanged between partners or used during the technology demonstration. Risk 
indicators resulting from the predictive analytics will not be exchanged between partners. As a result, a Privacy Impact 
Assessment will not be required. 


For the purpose of the technology demonstration, the cell phone application will not support all mobile operating systems. 


9. Assumptions 


e The technology platform used in the technical demonstration exercises will be provided by WorldReach in their 
performance lab and development facilities : 


+ All project team members will be available to perform their assigned tasks when required or suitable replacements 
will be identified. 

ə No real travellers or data will be part of the technology demonstration. Volunteers will be invited to participate in 
the demonstration using their ePassports. 


10. Constraints 


e All partners must work within the constraints of existing programs, policies and infrastructure, which will likely 
remain in place for at least 2-3 years. This includes the features of the ePassport, the eTA program, border kiosk 
technology, mobile devices, Integrated Primary Inspection Line (IPIL), and the GCMS used by IRCC and CBSA. 

e The mobile app can only be developed on the Android platform at this time for NFC access. 
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11. Related Projects 


e “Architecture and Assessment: Privacy Preserving Biometrically Secured Electronic Documents” (CSSP-2013- 
CD-1064): biometric encryption in the travel continuum and proposed the architecture which this prototype 
expands. 


PROJECT PLAN 
12. Schedule and Work Breakdown Structure 


12.1 Schedule: The Project started on 1 April 2017 and will be completed by 31 March 2018. 


For Contract(s) or MOA(s) established under this project, the project activities will start upon issuance of Contract or 


12.2 Work Breakdown Structure (WBS): The schedule shows the planned start and stop dates for each task in the WBS 
and appropriate milestones and/or delivery dates. The total Project period of performance is indicated in months 
from the start date through the completion date. 


S i ie 
cs | atl AO: 


initiation: 


- Contracts 
Prepare Charter with overview of | - Project kick-off meeting CBSA 
activities, resource requirements | - Go/No Go Decision for Execution WorldReach 
& roles of team members. Entrust 
UofO 


Apr/17 May/17 


cution `- 


nid 


May/17 


Initial Design/Technical Briefing: 

Partners collaborate to further 

define ConOps and high level - High-Level Design Report WorldReach 

design, including: Entrust 

- Project Briefing UofO 

- Point of View Face4 

- Process and Information 
Flows 

- Technology Blocks 


Technology Refinement: 


Implementation of 
enhancements to current 
platforms in preparation for four 
iterations of a prototype. The 
following technology areas will 


- Updated Concept of Operation CBSA Apr/17 


IRCC 
CBSA 
WorldReach 
Entrust 
UofO 

Face4 


Four prototypes to be produced with 
increasing degrees of fidelity: 


May/17 | Jan/18 


- Technology Refinement Fidelity 1 
(July 2017) 


- Technolog 


Refinement Fidelity 2 
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be covered in this phase: 


ePassport Interoperability 
PrivBio 

Collaborative Risk Engine 
Server & Traveller Risk Profile 
IRCC Predictive Analysis 
CBSA Predictive Analysis 
Universal Traveller Record 
(UTR) 

Face Recognition 
Identification 


Technology Integration: 


Design and simulate integration 
points in preparation to the 
prototypes. The following 
technology areas will be covered 
in this phase: 

- Mobile Apps (FASTER, 
Passport Renewal, E311 
Submission, MyCBSA) 
Operational Support Mobile 
Apps (Checkpoint Assistance, 
Mobile Enforcement) 

Kiosk 

Predictive Analytics and Risk 
Assessment (IRCC and 
CBSA) 

Collaborative Risk Engine & 
Policy Enforcement Points 
Backend Prototype Hamesses 
(e.g server integration 
FASTER to GCMS, 


Scenario Workshop: 


Drafting of scenarios for the 
prototypes aiming to closely 
simulate process/traffic flows 
through the different stages of 
the travel continuum. 


Initial Setup 
Based on the Logistics plan and 
identified scenarios, the team 
will configure the technical 
environment and eye 
p2is.2\Conduc SERRE A 


RS 


ET 


Fidelity 1 - Fidelity 1 Exercise Report 
The Ed scenarios will be 
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(September 2017) 


- Technology Refinement Fidelity 3 
(November 2017) 


- Technology Refinement Fidelity 4 
(January 2018) 


Brief integration reports for each of 
the four prototypes: 


Technology Refinement Fidelity 1 
(July 2017) 


Technology Refinement Fidelity 2 
(September 2017) 


Technology Refinement Fidelity 3 
(November 2017) 


Technology Refinement Fidelity 4 
(January 2018) 


Draft Scenario Report 

Exercise Logistics Plan Report 
Walkthrough Exercise Findings 
Report 

Project Review Committee 

Go/No Go Decision for Technology 
Exercises 


Exercise Technical Environment 
Ready 
Exercise Facility Ready 
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May/17 | Jan/18 


May/17 | Jun/17 
i} 
Jun/17 


IRCC 

CBSA 
WorldReach 
Entrust 
UofO 

Face4 


IRCC 

CBSA 
WorldReach 
Entrust 
UofO 

Face4 


IRCC 

CBSA 
WorldReach 
Entrust 
UofO 

| Face4 


Jun/17 


IRCC 
CBSA 
WorldReach 


Las ži 
Jul/17 Jul/17 


Page 9 of 17 


A-2018-09320-001085 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


Project #: CSSP-2016-CP-2276 UNCLASSIFIED 


executed using the configured Entrust 
facility and technical UofO 
environment. This first exercise 

primarily involves technology 

already available. 


Fidelity 2 - Fidelity 2 Exercise Report 
Based on the observation of the 
Fidelity 1 exercise, the 
scenarios, facility and technical 
environments will be adjusted. 
A new release of the technical 
components will be installed. 
The updated scenarios will be 
executed using the refined 
facility and technical 
environment. 


WorldReach 
Entrust 
UofO 

Face4 


Fidelity 3 - Fidelity 3 Exercise Report nr Nov/17 
Based on the observation of the WorldReach 

Fidelity 2 exercise, the Entrust 

scenarios, facility and technical UofO 

environments will be adjusted. Face4 

A new release of the technical 

components will be installed. 

The updated scenarios will be 

executed using the refined 

facility and technical 


executed using the refined 
facility and technical 

environment. 

Jan/18 

environment. 

Evaluation Final Raport en Jan/18 | Feb/18 
on Re eus present - Evaluation Report WorldReach 
Refinement THA - Technology Dissemination Plan Entrust 
nant Bonno ogy - Outline of a potential pilot UofO 
Integration and Fidelity Face4 
Exercises, including 
recommendation on next steps. 
Project Close Out | IRCC 
Assess project deliverables - CSSP Project Completion Report | CBSA Feb/18 | Mar/18 
against objectives and to document that all deliverables WorldReach 
investments. met their criteria and suggest Entrust 
potential technology transition UofO 
options/follow-up items. 
- Close-Out Presentation 
- Project Review Committee Meeting 
Minutes 


Fidelity 4 - Fidelity 4 Exercise Report ee Dec/17 
Based on the observation of the WorldReach 
Fidelity 3 exercise, the Entrust 
scenarios, facility and technical UofO 
environments will be adjusted. Face4 
A new release of the technical 
components will be installed. 
The updated scenarios will be 
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The following activities will be 


performed to successfully 


manage the project: - August 2017 


- Consolidate/report on project - November 2017 
- February 2018 


financials to ensure proper 
communication to 
stakeholders. 

- Evaluate and track risks 
according to risk management 
plan. 

- Monitor project requirements 
against agreed scope. 

- Document changes for 
approval. 

- Manage milestones and time 
to completion to identify and 
communicate impacts to 
baseline schedule. 


Year End Financial Report 


Progress Reports for the CSSP in: 
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WorldReach 
Entrust 
UofO 


The Project will provide DRDC CSS with the following outputs (templates to be provided by DRDC CSS): 


Signed Project Charter 

Quad Chart 

Progress Reports 

YE Financial Report (for other government departments) 
Project Completion Report 


13. Budget 


TOTAL | 
$237,000 


14 


1105 $33,333 $33,333 


CBSA Equipment & ES $110,000 $110,000 
Supplies 


| Toa | |s12e0s23 | $1,296,523 | 
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13.2 


In-Kind Co-Investment 


H 


a À 


$344,000 
$570,000 
$175,000 


GAT 
ae q 
3 
? 
1 
3 
3 
3 


| Tota || #sro2o | $1,416,800 | $1,496,000 


14. Resource Requirements 


0.1 


14.2 Facilities and Equipment 


The following equipment and facilities will be required for the Project: 
- CBSA facility in Rigaud for the final demonstration (TBC). 
- A performance lab will be required throughout the duration of the project. 


Insufficient number of 
volunteers for the 
demonstration 


Insufficient variety of 
passports to test the 
broad applicability of the 
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Regulatory/policy 
framework for border 
management unready 
for Chain of Trust 
Diverging interests of 
commercial partners 
related to IP 


IRCC or CBSA 
programs not ready to 
assist or implement 
technolog 


Lack of available low- 
cost facilities for the 
demonstration 


eTA data is of L H Technical 
insufficient quality to 

enable the development 

of predictive models 

Predictive models are =} L H Technical 
incapable of finding 

patterns in the data 


Note 1: (H/M/L) = high, medium or low 


Note 2: Examples of "Risk Type” include: financial, schedule, scope, technical 


16. Evaluation of Project Success 


UNCLASSIFIED 


A government policy team at IRCC is already 
analyzing legal and privacy factors to be 
considered in the development of the 
technology/concept. 


Commercial partners to meet while Charter is 
being drafted to reach agreement or revise 
their level of participation 


IRCC and CBSA stakeholders have initial buy- 
in and will be given regular project updates. 
Stakeholders will also be involved in the Project 
Exercises to ensure Project momentum. 
Combination of IRCC and CBSA resources 
allow sufficient alternatives. 


The eTA Operations team is working closely 
with the GCMS team to ensure that data quality 
is very high, notably through a web-based 
application form containing mandatory fields 
and by replacing open text fields with drop- 
down menus. 

IRCC can hire very experienced consultants to 
complement its extensive in-house expertise on 
model development. Positive results obtained 
for other lines of business will be leveraged for 
eTA. 


The Chain of Trust project will be evaluated based on the following criteria: 

1. The smartphone is able to capture all the information from an E311 Declaration Form. 

2. The smartphone is able to submit E311 information in a secure manner 

3. The smartphone is able to demonstrate how to populate a mobile ePassport renewal application with relevant 


passport data. 
4. 


5, The IRCC predictive model is able to triage eTA applicants into high, medium and low risk groups. 
6. The CBSA predictive model is able to incorporate IRCC’s risk assessment and new data collected during the trip to 


triage travellers into high, medium and low risk groups. 


7. Scenarios are conducted to test the project concepts which includes: 
a. a sequence of checkpoints and information flows to support continuous risk assessment (ex. self-serve 
enrolment, adjudication of application information, pre-boarding checkpoint, Primary Inspection, 


Mobile, Egress) 


b. Volunteer users acting in the roles of travellers and operational personnel 


c. Reporting on traveller throughput statistics 


8. A document which describes the integrated system and the results of each scenario test. 


17. Procurement Strategy 


This project was approved through the call for proposal process. DRDC CSS will transfer funds to IRCC and CBSA. IRCC 
will establish a contract with WerldReach Software, while CBSA will establish contracts with the University of Ottawa, 
Entrust Datacard and Face4 Systems. Payment will be firm fixed price with milestone payments. 
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18. Transition Strategy 


The prototype will be delivered in collaboration with the federal operational stakeholders. This will ensure a 
Strategic/legislative fit within Canada’s overall borders/immigration, identity management, law enforcement/intelligence 
Strategies. 


The reports and non-confidential deliverables of this prototype will be made available within Government and the partners 
in the project who undertake to assist in the promotion of the results and leverage the knowledge gained via the prototype. 
The project team will prepare next step recommendations as part of the technology dissemination plan deliverable to be 
prepared at the end of the project. 


Demonstrated capabilities of this prototype will create leave-behind knowledge, technology, policy inputs and operational 
benefits to IRCC, CBSA and other stakeholders/partners during and after the project. Once confidence in the prototype 
has been established, the team could recommend a pilot, which could include: 


e Strategic approval for the pilot by lead departments; 

e Costing of the pilot and identification of funding sources; 

e Technological inputs to immigration/border strategies, plans etc.; 

e Further research/development required to take the pilot forward; 

e Formal Threat Risk Assessments and Privacy Impact Assessments by relevant stakeholders; 
e Policy Development and Approval processes; 

e Contribution to international presentations such as ICAO and industry forums; 

e Procurement of key components and provision of detailed proposals for implementation. 


With the involvement of Treasury Board and the lead departments, additional opportunities for other uses of the 
science/technology could be identified for secure identification and authentication in a privacy respecting manner such as: 


Virtual Passport 

“Know your customer/citizen” tests; 
Trusted traveller enrolment; 

Other government e-services 


Consideration will be given to invite representatives of the Five Country Conference (FCC) to have observer assigned to 
the project who may observe key exercises or volunteer to participate in testing. This will help to align approaches across 
the FCC and facilitate information sharing and best practices. 


PROJECT SUPPORT 


19. Project Organization 


Roll aime |. : Organizatio {Phone Numbe ‘Email: Addres: i 

Mark Williamson DRDC CSS 613-944-8195 Mark.Wiliamson@forces.gc.ca 
Leader i 

Brian Greene DRDC CSS 613-943-0751 Brian.Greene@forces.gc.ca 
Director 

Simon Cardinal IRCC 613-437-6289 Simon.Cardinal@cic.gc.ca 
Champion 

IRCC 613-437-5556 steven.gonzalez@cic.gc.ca 
Manager 
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| Contracting | Heather Wilson PWGSC 


| Deputy Tony Mungham CBSA 
| Project 


| Manager 


| Technical 
Authority 
Principal 

Investigator 


David Bissessar CBSA 


873-469-4791 


ER NN 


heather. wilson@ipsac- 
pwgsc.gc.ca 


Tony. Mungham@cbsa-asfe.gc.ca | 


613-952-3163 


David Bissessar@cbhsa- 
asfc.gc.ca 


WorldReach 
| Software 


Industry 
| Partner 


613-742-6482 


industry 
Partner 


i Entrust 
Datacard 


Industry 
Partner 


Face4 Systems 


University 
Liaison 


University of Ottawa 


613-270- 


613-656-3709 


enirustdatacard.com 


@worldreach.com 


& 


@iacedsystems.com 


613-562-5800 
ext. 


20. Relevant Agreements/Arrangements 


@uottawa.ca 


intellectual Property (IP) will be managed in accordance with the General Conditions ~ Research & Development 2040 
and specific clauses. These will be outlined in the Contract General Conditions. 


Within the CSSP Program, Canada is utilizing the Commercialization in Canada clause to manage the Foreground 
information. Within the clause, the contractor will own any Foreground IP that is created as part of that contract or 


memorandum of agreement. 


Transfers of funds to CBSA will be managed in accordance with the terms and conditions set out in the Public Security 
Science & Technology (PSTP) MOU between the Department of National Defence and various other Federal 
Departments and Agencies, including the Canada Border Services Agency, which is a signatory to the MOU (DND 


identification Number: 2008040021: 


Transfers of funds to IRCC will be managed in accordance with the terms and conditions set out in the Interdepartmental 
Letter of Arrangement (ILOA) between the Department of National Defence and Immigration, Refugees and Citizenship 


Canada. 


21. Other Information 


IRCC interdepartmental Settlement (1S) codes: 
Department Code : 085 
Reference Code: 3910 
Organisation Code: 9000 


CBSA Interdepartmental Setilement US) codes: 
Department Code : 085 
Reference Code: TBC 
Organisation Code : 3910 
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22. Communications 


DRDC CSS reserves the right to disclose and/or use information from projects for which it provides funding, including the 
publication of a short project description on science.gc.ca/cssp. Should you have any concems about project information 
being posted on the website, please discuss them with your DRDC CSS Project Director during the charter development 
stage. Additionally, the use and publication of information related to this Project will require review by DRDC CSS to 
ensure proper acknowledgement of DRDC CSS/CSSP support/contribution. In order to achieve this, Project participants 
will provide copies of any reports, articles, or publications to DRDC CSS at least 30 days, where possible, prior to any 
release, distribution or planned publication. No public announcements concerning this Project will be made without 
providing advance notification to DRDC CSS. Project Managers and Directors should ensure that communications are 
executed through a coordinated approach that favours the development of joint products and activities. DRDC CSS must 
receive a copy of all reports generated under the Project. 

23. Publication/Security/Controlled Goods/Distribution 


The highest level of security to be applied to information generated or exchanged by or in this Project will be Protected B; 
a Security Requirements Check List (SRCL) will be required. 


The distribution of information resulting from this Project is Limited. Where limited, distribution is limited to the following: 
federal government departments and agencies, as well as project partners. 

24. Project/Charter Amendments 

Any revision to or extension of this Project will be made through a formal amendment to this Project Charter, as concurred 
in writing by the participants. 

25. Project Termination 


DRDC CSS, in consultation with the Lead Government Department, will make recommendations regarding the termination 
of a Project, whose decision will be final. 


Some conditions that may lead to termination could include: 
e Charter has not been signed by project participants within three (3) months of project award. 
e Deliverables / Milestones not met (non-performance). 
e Forecast inability to deliver (for example, key personnel have left the department or Project). 
e Failure of a participant to meet contractual obligations or PSTP MOU reporting requirements. 
e Change in CSSP strategic priorities or science and technology requirements. 


“e Lead Government Department no longer leading the research study. 
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Chain of Trust (CoT) ai 
Initial Design Workshop Outcomes 


Date: June 1, 2017 Reference: CoT201 70601 


Background: 

The Science and Engineering Directorate (S&E) at the Canada Border Services Agency (CBSA) led a five- 
day workshop from May 8th to May 12th, 2017 to determine the high-level design of the prototype for 
the Chain of Trust: Collaborative Risk Assessment for Air Traveller Programs project. This is a project 
under the Canadian Safety and Security Program (CSSP) - CSSP-2016-CP-2276. The workshop was based 
on the principles of design thinking and rapid prototyping. Annex 1 lists the project resources who 
attended the workshop. 


The Chain of Trust project addresses the CSSP Challenge 14 from the 2016 CSSP Call for Proposals: 
Enhance targeting solutions that make use of all available data, including electronic traveller data and 
biometrics, for the purposes of identifying low-risk travellers. The project focuses on international 
travellers coming to Canada using the Electronic Travel Program (eTA) and returning Canadians (both via 
air). The project contains three key areas of activity: 


e Collection of high quality data earlier in the travel continuum. 


This data collection will be possible through 
enhancements to the FASTER PrivBio smartphone app (developed through a previous CSSP 
Project), which will also include the ability to complete the E311 Declaration Form online and 
will support a broader range of devices. 


e Consolidating available data inputs collected 
throughout the travel continuum into a new transaction record (the Traveller Risk Profile or TRP) 
to enhance resolution of the traveller identity and provide inputs into risk assessments. Each 
TRP will update the Traveller Passage Record (TPR) containing both specific data from the TRP 
record and historical traveller data. 


e A privacy-respecting, collaborative predictive analytics approach by IRCC and CBSA leveraging 
historical traveller information and trip-related electronic data collected throughout the journey 
Collaborative predictive analysis will be updated as new information about the traveller is 
collected throughout the continuum to inform the risk triage of travellers so that low-risk 
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travellers benefit from expedited processing while high-risk individuals are flagged for greater 


Services frontaliers 


scrutiny. 


The Chain of Trust (CoT) project brings together Immigration Refugee and Citizenship Canada (IRCC), the 
Canada Border Services Agency (CBSA), Industry (WorldReach Software, Face 4 Systems, Entrust 
Datacard), as well as academia (University of Ottawa) in a thought-leading approach to apply 
knowledge/technologies for collaborative risk assessment and establishment of the traveller identity 
throughout the travel continuum. 


The work follows an agile and iterative methodology. The work commences with this baseline design 
workshop and will be followed by four exercises. Each exercise will involve design refinement, scenario 
development, technical development, and the execution of the exercise, followed by results assessment 
to inform the next exercise. 


Exercises are in in a staged environment and include all project partners. The exercises will not deal with 
real travel transactions. Rather, they will make use of actors using mock data to represent travellers, 
border and immigration resources. 


The project will end with an evaluation report that will describe what worked well and what did not 
work, and make recommendations for a future pilot that would exercise the concept in a real travel 
environment. 


Prototype Stakeholders and Their Needs: 

There are a number of stakeholders that will benefit from the concepts related to CoT, or who have an 
interest in the concept. The prototype will focus on meeting the direct needs of the IRCC eTA and 
Passport programs as well the CBSA programs related to air travel by informing the risk triage of 
travellers so that low-risk travellers benefit from expedited processing while unknown and/or high-risk 
travellers are flagged for greater scrutiny. However, the pilot will also take into consideration the 
benefits and interactions of other stakeholders involved in the air travel continuum. 


The Chain of Trust Prototype will focus on the following stakeholders: 


International air traveller requiring an eTA: This is a traveller from a foreign country who wishes to 
come to travel Canada and needs a valid eTA in order to travel to Canada 


Canadian traveller returning to Canada via air: This is a Canadian traveller who wishes to return to 
Canada and needs a valid Canadian Passport in order to travel to Canada 


Airline: This is a commercial organization that provides the air travel to the traveller, The Airline must 
sell the traveller a ticket, must validate that the traveller has a valid Passport or eTA {if required), and 
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must send API/PNR info to the CBSA at set intervals in the travel continuum (72Hrs, 48Hrs, 24Hrs, and 
Wheels Up). 


Services frontaliers 


IRCC ETA Review: This is a group at the IRCC that reviews applications for an ETA and either approve or 
denies the issuance of an ETA. The IRCC ETA Review group needs sufficient information in order to 
review an ETA application 


IRCC Passport Renewal: This is a group at the IRCC the reviews renewals for passports and either 
approves or denies the issuance of a passport. The IRCC Passport Renewal group needs sufficient 
information in order to review a Passport Renewal. 


CBSA Client Service: This is a group at the CBSA that maintains an electronic conversation with a 
traveller in order to establish a trusted rapport with the traveller and promote compliance. The CBSA 
client service group needs a method to communicate with the traveller in a privacy preserving manner. 


CBSA Targeting: This is a group at the CBSA that reviews information on a traveller in order to 
determine if a traveller requires administrative facilitation or enforcement examination. The CBSA 
Targeting group needs sufficient traveller and related information to perform a risk assessment on a 
traveller or group of travellers. 


CBSA POE Administration: This a group at the CBSA that manages the deployment of CBSA resources at 
an international airport. This group can have a scope that is local to an airport and/or national across the 
country. The CBSA POE Administration need sufficient near real-time information related to the 
movement and processing of travellers in order to effectively deploy CBSA resources. 


The CBSA Mobile Enforcement Officer: This is a group at the CBSA that moves around the various travel 
zones at an airport and interacts with travellers in order to identify possible non-compliance. The CBSA 
Mobile Enforcement Officers need sufficient near real-time traveller information and the ability to 
communicate with each other and a command centre in order to detect and interdict non-compliance. 


The CBSA Mobile Enforcement Command: This is a group at the CBSA that monitors through overt 
surveillance the movement and behaviour of travellers in order to identify travellers that require an 
interaction with a Roving Officer, and directs a roving officer to the traveller. The CBSA Mobile 
Enforcement Command needs to see the movement and behaviour of the travellers, have sufficient 
information to develop indicators of possible non-compliance, and be able to communicate remotely 
and efficiently with the roving officers 
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Design Point of Views: 

A point-of-view (POV) is the reframing of the design challenge into an actionable problem statement 
that will launch the design team into generative ideation. The POV creates How-Might-We questions 
that facilitate the designers to ideate in a directed manner. The POV captures the design vision from 
which the design team discovers and articulates the meaningful challenge. 


Data Collection: 
How might the design team collect data about a traveller as early as possible and as often as 
possible throughout the travel continuum? 


Data Consolidation: 
How might the design team continually consolidate the collected traveller data to enhance the 
resolution of the traveller identity and provide inputs into risk assessments? 


Identify and Expedite Low Risk Travellers: 

How might the design team implement systems to identify low-risk travellers who will benefit 
from expedited/self-service processing while high-risk travellers are flagged for examination by 
IRCC and CBSA officers. 


Prototype Zones and Stakeholder Interactions: 

The Chain of Trust prototype will be divided into discrete travel zones. Each zone is a boundary that 
describes either a logical aspect of the system and/or physical location of a traveller during an 
interaction with the Chain of Trust system(s). Below are the travel zones that will be considered during 
the prototyping of the Chain of Trust System. Each zone is described along with an overview of the 
interactions that take place: 


Intent to Travel Zone 


A traveller shows an intent to travel by either applying for an ETA, renewing a Passport, or initiating a conversation with 
the CBSA. This zone is a logical zone and does not have a defined physical location. 


Traveller’s Perspective IRCC/CBSA Perspective 
Trigger: Trigger: 
e The traveller intends to travel ə The traveller has indicated an intent to travel 
Actions: Actions: 
e Renew Passport (Canadians only - if applicable) e —IRCC ETA Review/IRCC Passport Renewal 
e  GeteTA (if applicable) receives application for eTA/passport renewal 
e Download the e-Declaration App (optional) e  IRCCIRCC ETA Review/IRCC Passport Renewal 
e Start conversation with CBSA/IRCC (optional) provides or denies eTA/passport renewal 


e CBSA Client Service responds to initiative 
conversation 


o Pushes info to promote compliance 
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Buy a Ticket Zone 


A traveller initiates an interaction with an airline through purchasing a ticket and the airline initiates API data sharing with 
the CBSA. This zone is a logical zone and does not have a defined physical location. 


Traveller’s Perspective CBSA Perspective 

Trigger: Trigger: 

e The traveller buys a ticket to Canada 
Actions: 


e Commits to travel 


e 72 hours before the traveller is going to travel 
Actions: 

e CBSA Targeting receives partial API from the 
e Provides personal information and payment airline company (no PNR details) 
e May check-in if within 24 hours (optional) e CBSA risk assessment starts 
o  Ifso, the traveller would provide 

information about luggage and passport | Trigger: 
number e 48 hours before the traveller is going to travel 
o The traveller may make a pre e- Actions: 
Declaration (optional) e CBSA Targeting receives the updated partial API 
e Downloads the e-Declaration App 


. Continues conversation with CBSA/IRCC 


from the airline company {no PNR details) 
° CBSA risk assessment continues 


Trigger: 
e 24 hours before the traveller is going to travel 
Actions: 
e CBSA Targeting receives updated API and partial 
PNR from the airline company 
e CBSA risk assessment continues 


Pre Departure Zone 


A traveller arrives at an airport and goes through the departure process at the airport. For the purposes of the Chain of 
Trust Prototype, this includes boarding the aircraft. This is a logical zone and is physically at a departure airport. 


Traveller’s Perspective CBSA Perspective 
Trigger: Trigger: 
e The traveller arrives at the airport e The traveller arrives at the airport 
Actions: Actions: 
e  Checks-in luggage e The airline checks the traveller’s document 
e Clears security e The airline accepts luggage 
e Arrives at the gate e CBSA risk assessment continues 
e Shows travel documentation 
e Boards plane 


e Downloads the e-Declaration App (optional) 


e CBSA commits travel as passage (program policy 
commit) 
e Airline Checks and validates passenger 
o The airline checks CBSA for board/no- 
board, 
O The airline validates boarding (multiple 
is possible) 


è  Makes/updates the pre e-Declaration (optional) 
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Pre Arrival Zone 


route (wheels-up). 
Traveller’s Perspective 
Trigger: 
e Wheels Up — Flight headed to Canada. 
Actions: 
e Completes e-Declaration App (optional) 
e Submits e-Declaration (optional) 


Trigger: 
e 
Actions: 


The traveller is on an aircraft and is on route to Canada. This is a logical zone and is physically represented by an aircraft on 


CBSA Perspective 


Wheels Up — Flight headed to Canada. 


CBSA Targeting receives final API and full PNR from the 
airline company 

CBSA POE Admin plans for traveller flow 

CBSA makes recommendation for referrals based on: 


CBSA POE Admin planning 

Reviews indicators 

Prepares response 

Gets updated arrival times from airline 
Updates Dashboard for Ops 


oO Oo 0 0 


Arrival Zone 


Traveller’s Perspective 
Trigger: 
e The traveller disembarks the airplane 
(traveller may indicate arrival) 
Actions: 
e Completes e-Declaration (optional) 
e = Make finals e-declaration though 
mobile (optional) 
e Gets recommendations for next steps 
(optional) 
è Gets next step from CBSA (where to 
present themselves) 


Trigger: 


Actions: 


The traveller disembarks an aircraft. This is a logical zone and is physically represented by an aircraft arriving at a terminal, 
passengers disembarking, and moving towards the primary zone. 


CBSA Perspective 


The traveller disembarks the airplane (traveller may indicate 
arrival) 


CBSA Client Service sends the traveller a wait-time notice for 
the kiosk 

CBSA POE Admin updates arrival instance 

CBSA accepts final declaration and enrols the traveller into 
passenger flow control (optional — if ID was validated by 
CBSA previously) 

CBSA risk assessment continues 

CBSA POE Admin receives risk assessment details 

CBSA tells traveller next step 

CBSA ME Command and CBSA Rover observe behaviours 
CBSA ME gets indicators 

CBSA ME intercepts known threats 

CBSA ME and ME Command updates risk assessment 
attributes 
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Primary Zone 

A traveller presents themselves to CBSA and makes a final declaration. This is logical zone and for the Chain of Trust 
prototype it is physically represented at a Primary Kiosk in the arrivals hall of the CBSA or through a smartphone 
transaction in either the arrival area of Canadian airport or the arrival area of the CBSA. 


Traveller’s Perspective CBSA Perspective 


Trigger: 
e The traveller arrives at the kiosk or initiates 
primary processing on a smartphone 
Actions: 


Trigger: 
e The traveller arrives at the kiosk or initiates 
primary processing on a smartphone 
e presents face Actions: 
e scans e-travel docs (optional for travellers with low 
level of assurance) 
e — makes final declaration 
e gets recommendations for next step 


e CBSA POE admin monitors and adapts traveller 
flow based on volume 

e CBSA validates the traveller’s documentation 
and ID 

e CBSA enrols the traveller into passenger flow 
control 

e CBSA accepts final declaration and enrols the 
traveller into passenger flow control (optional) 

e CBSA risk assessment continues 

e CBSA POE Admin receives risk assessment details 

e CBSA ME and ME Command observe behaviours 

e = CBSA ME gets indicators 

e CBSA ME intercepts known threats 

e CBSA ME and ME Command updates risk 
assessment attributes 

e CBSA tells traveller next step 


Intermediate Zone 
The traveller completes any required administration transactions that do not require examination of goods or immigration 
documents (such as the payment of duties and taxes). This is a logical zone and is physically either at a kiosk (self-service) 

or service counter (border officer/clerk assistance). 


Traveller’s Perspective CBSA Perspective 
Trigger: Trigger: 
e The traveller is sent to the Intermediate zone e The traveller goes to the Intermediate zone 
Actions: Actions: 
e Complete non immigration administrative tasks e CBSA validates traveller location with flow 
(pay taxes, duties, fees; get permits; etc.) control system 
e Complete immigration tasks that do not require e CBSA processes administrative task 
examination e CBSA risk assessment continues 
e CBSA POE Admin receives risk assessment details 


e CBSA ME and ME Command observe behaviours 

e CBSA ME gets indicators 

e CBSA ME intercepts known threats 

e CBSA ME and ME Command updates risk 
assessment attributes 

e CBSA tells traveller next step 
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Secondary Zone 


A traveller is examined by a border services officer for either customs and/or immigration purposes. This is logical zone, and will 
not be physically represented in the chain of trust prototype. 


Traveller’s Perspective CBSA Perspective 
Trigger: Trigger: 
e The traveller is sent to Secondary e The traveller arrives at Secondary 
Actions: Actions: 
e Answer questions e CBSA validates traveller location with flow control 
e Present goods system 
e Present documents e CBSA gets the reason why they have been sent to 
Secondary 


e CBSA notes the results of the exam 
e CBSA updates the risk assessment 
e CBSA POE Admin adapts to traveller flow 


Egress Zone(s) 


A traveller exits the CBSA processing area and enters Canada. This is both logical zone and physical zone that is represented by 
the final control point and/or one or more verification point(s) in the CBSA processing area. 


Traveller’s Perspective CBSA Perspective 
Trigger: Trigger: 
e The traveller is trying to exit the CBSA processing e The traveller arrives at the Egress location 
area Output: 
Actions: e CBSA POE Admin monitors and adapts traveller flow 
e Present themselves e CBSA validates traveller location with flow control 
e Follow directions (they are released or referred back system 
to a point in the process) e CBSA ME gets indicators 


e CBSA ME intercepts known threats 

e CBSA ME and ME Command observe behaviours 

e CBSA ME and ME Command updates risk 

assessment attributes 

° CBSA risk assessment continues 

e  CBSA tells the traveller next step: 
o “Refer” — the traveller is sent to Secondary 
o “Release” — the traveller can leave the 

airport 

o If released — CBSA closes the passage 
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The prototype design is starting with the concept that there are three overarching policy themes: 
immigration concerns, customs concerns, and overall public safety concerns. Each policy theme 
comprises of a set of policies (policy map) that are determined by the characteristics of the traveller. For 
example, a returning Canadian would have a set of policies that are different from a foreign national 
that requires an ETA. A set of policies is made of individual policies that are essentially individual 
questions that contribute to understanding the characteristics of a traveller. The result of a policy can 
come from many sources such as testing a rule With given parameters, the results of a database search, 
or the results of predicative analytics. The result of a policy can also drive branching policies that further 
clarify the characteristics of a traveller. There are two types of policies: 


1) Scoring policies work together to establish the characteristics of risk associated to an instance of 
a traveller moving through the border. Each scoring policy has a score to indicate the level of 
concern, and each score can result in a state of either 


The thresholds to change the policy 
state from one result to another is operationally dependent. 


2) Trigger policies are used to drive the workflow. These are typically if-then logic and trigger the 
next action within the prototype system. 


Thus, each traveller would have a cluster associated to them and this cluster is based on a dynamic set 
of policies that evolves over time as the characteristics of a traveller are assessed and/or determined. 


Figure 1: Example of Policy Questionn 
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Predictive Analytics 


The heart of the Chain of Trust prototype is predictive analytics. The predictive analytics uses machine 
learning to identify patterns that can be used to inform downstream processes. This prototype will have 
two distinct predictive analytic engines: one for the IRCC related to the issuance of ETAs, and the other 
for the CBSA that will be related to the indication of non-compliance of travellers. Both engines are 
based on organizational specific program policy coverage, so the results will be shared through the risk 
assessment collaboration tools that will ensure programs policy coverage and limits are respected. 


Risk Assessment and Workflow Orchestrator 


The ability to manage the individual policies, the policy maps, and to execute the risk assessment in a 
collaborative way is accomplished by an orchestrator that will manage the risk assessment and the 
dynamic rules/risk based workflow of a traveller. 


Managing Privacy through the use of Renewable Biometric References 

The Chain of Trust prototype will be linking travel documents, travel credentials, and risk characteristics 
to a traveller through the use of biometrics, but this will be done by creating a Renewable Biometric 
Reference (RBR). A RBR is derived and verified from a live biometric through the use of cryptography 
without storing the actual biometric and/or biometric template. The biometric cannot be reconstructed 
from the RBR but the RBR can be verified against a live biometric. 


The use of RBRs as opposed to traditional biometrics allows the CBSA and IRCC to link people to travel 
credentials and risk characteristics without storing and managing the private biometric information of 
tens of millions of travellers. It removes the risk of “loosing” biometric information if a system is 
attacked or compromised by providing a mechanism to revoke/cancel and regenerate RBRs all while 
never storing the actual biometric information itself. 


Face Recognition Based Flow Control 

The prototype will be using and testing the effectiveness Face Recognition (FR) on the move to manage 
the flow of travellers within the CBSA processing area at an airport. The traveller will be enrolled into 
the FR flow control system during the primary zone interaction, and choke points with FR verification 
will be used to manage the flow of travellers as they move from one zone to another zone. The most 
important zone to ensure a high level of integrity is the final egress zone. In this zone, the traveller is 
either directed back for further processing/examination or is allowed to enter Canada. Once the 
Traveller has completed border processing (e.g.: is allowed to enter Canada), the traveller is unenrolled 
from the FR flow control system. 


Vision Box Orchestra 

During the design sessions the idea of using existing software to help visualise and manage the flow of 
travellers within the CBSA processing area was discussed. A third party vendor named Vision Box has a 
software tool called Orchestra. The project will look into using Orchestra in order to capitalize on 
existing capabilities, thus allowing the project to focus on new and untested platforms. 
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Prototype System Interactions: 
The prototype system interaction is represented as a block diagram. The diagram below shows the 
major technology blocks and indicates the links between the individual blocks. 


Figure 2: Chain of Trust Block Diagram 


Prototype Scenario Storyboards: 

The prototype is an end-to-end concept that will be tested through a set of exercises that are based ona 
number of stories. Each of the stories will test various aspects of the system while driving a variety of 
outputs. If the output occurs as predicted, the design is working as planned. However, if the output was 
unpredicted, it may be necessary to review and update aspects of the design 


Stories are an effective means to coordinate a common purpose in scope. They ensure that all project 
partners are working towards the same desired outcome allowing independent work to align to a single 
goal. The Chain of Trust design will be initially tested based on the four baseline scenarios. However, it is 
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likely that additional stories will be created over time in order to ensure certain aspects of the prototype 
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are tested. 


The four baseline scenario storyboards that Were created during the initial design workshop are 
described below. These stories will be elaborated into a narrative and reviewed by all project partners 
before the first exercise as the story narrative is core to the planning of each exercise. 


Scenario #1 — Broken RBR: 

Canadian traveller arrives 

Submits eDeclaration 

Overlimit > traveller directed to Intermediate 

Traveller pays fees at intermediate kiosk, but the RBR fails ~ traveller directed to Secondary 
(Back office alerted — Rover dispatched to find traveller) 

Rover finds traveller, checks face 


Oy ON a ND ak 


Scenario #2 — Missing bag: 

Canadian traveller’s API indicates they checked 5 bags 

Traveller submits eDeclaration and indicates 5 bags 

Traveller only picks up 3 bags at baggage carousel 

Traveller goes to airline to report 2 lost bags (fills e-A23 form) 

Traveller heads to Egress 

Face Recognition workflow links the traveller to number of bags 

Information displayed on screen for an officer for verification {could be non-BSO) 
Send traveller to Egress Rover. 

OK 


£000 OY UP OU 


Scenario #3 — Time delay: 

Foreign national granted eTA 

Traveller generates RBR 

Traveller checks-in (Trip Record starts) 

eTA Predictive Analytics hit 

Wheels Up (Trip Record, Risk Assessment products) 
eDeclaration completed on plane 

Arrival 

Traveller submits eDeclaration (validate RBR) 


AO OOF SH ON UT RIRES 


PPR 
NP 9 


(geo-locate by wifi) 

. Time analysis triggers 

13. , 

14. Rover dispatched to intercept traveller in Arrival hall 
15. Rover interviews traveller 

16. Primary Kiosk 

17. Egress 
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Scenario #4 — Passport Renewal: 
1. Canadian traveller get mobile app for entering Canada 


2. Traveller creates “My Profile” (RBR) 

3. App indicates to the traveller that their passport is close to expire date. 
4. Traveller poses a question to CBSA on the mobile app 

5. CBSA answers the question (provides “Did You Know?” trivia) 

6. Traveller buys a ticket 

7. Allow trip history to be viewed (passport stamp) 

8. Check in (e-check in) 

9. Get passport info — flag passport expiry date to traveller 

10. In flight 


a. Traveller completes eDec 

b. Traveller buys duty free items 

c. Traveller updates eDec 
11. Arrival 

a. Traveller submits eDec 

b. App reminds traveller that passport is about to expire 

c. App invites traveller to start passport renewal process on mobile 
12. Primary Kiosk 


Possible Missing Stories: 
Only four baseline storyboards were created during the initial design workshop. It may be of interest 
to develop stories related to: 
e Back office (CBSA Operations) in terms of dynamically changing the workflow based on Ops 
e flags to Secondary needed to be tested (Inadmissible Person, Suspected Smuggling) 
e Groups of travellers 
o associated travellers 
o disease 


Page 13 of 16 


PROTECTION + SERVICE + INTEGRITY 


A-2018-09320-001111 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


Border Services $ 


Prototype Partner Responsibilities 
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To ensure the success of the project, each partner is responsible for a specific aspect of the 
prototype. In some cases, there will be shared responsibilities and ongoing/close collaboration 
must exist in order to reduce the duplication of work. The responsibilities are listed below: 


o IRCC 
o Manage Project (Project Lead) 
o Predictive Analytics on ETA 


o Exercise Coordination 
o Exercise site fit-up 
o Operational apps (UI) 
=" Kiosks > outsource to Identos 
= Rover and Egress > BASD 
= NTC -Blackbox 
= Int app 
= Spec serv app 


o WorldReach 
o Host platform, 
o Traveller mobile app(s) 
o Mobile data collected from traveller 
o  Visionbox dashboard 


o Entrust 
o The “orchestrator” — policy based risk assessment and workflow 
o Supporting databases for orchestrator 
o Trip records 


o University of Ottawa 
o RBR 
o Predictive Analytics for CBSA (general > look for “doubt” ) 


o Face4 
o Face recognition workflow, 
o Manage traveller flow (execute with integrity check), 
o Visionbox dashboard. 
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Next Steps: 
The initial design workshop provides a foundation upon which the exercises will be built. The following 
next steps will evolve the project towards executing the testing exercises: 


1) Review this report across all project partners and agree on the foundational design 

2) Develop the narrative around the four baseline scenario stories to clarify the objectives ofthe 
exercises. 

3) Determine the policy questions and policy maps needed by the Risk Assessment/Workflow 
Orchestrator that will bring the four baseline scenarios to life in the upcoming exercises. 

4) Plan and execute the first exercise in order to flush out the elements of interest within the Chain 
of Trust concepts, and the details related to the level of development required for integration 
and testing of the elements. 

5) Develop measurements that will articulate the level to which the elements ofinterest in the 
chain of trust project work or don’t work. 

6) Establish the various mini design teams and the rhythm of work required to execute the 
exercises 
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Annex 1: List of project resources who participated in the workshop: 


[Name | Organization o O 
Barrett, Naomi Canada Border Services Agency 


Bissessar, David Canada Border Services Agency 


Galton, Eric Canada Border Services Agency 


Léonard, Marc (Lab) Canada Border Services Agency 


Mungham, Tony Canada Border Services Agency 
Wong, Jonathan Canada Border Services Agency 


Anderson, Jarek Canada Border Services Agency 


Miovski, Vasil Canada Border Services Agency 
Sanderson, Paul Canada Border Services Agency 


Tolmie, Michael Canada Border Services Agency 


El-taki, Moe Canada Border Services Agency 


Haymes.Michael Immigration Refugee and Citizenship Canada 
University of Ottawa 


University of Ottawa 
University of Ottawa 


Face4 Systems Inc. 


Face4 Systems Inc. 
WorldReach 


WorldReach 
WorldReach 
Entrust 
Entrust 


Page 16 of 16 
PROTECTION + SERVICE + INTEGRITY 


A-2018-09320-001114 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


s.19(1) 
Canada Border Agence des services 
Services Agency frontaliers du Canada 


Border Services 


Chain of Trust (CoT) 
Prototype 1 - Outcomes and Notes 


Services frontaliers 


Date: August 8th, 2017 Reference: CoT20170808 | 


Prototype 1: 

The Chain of Trust project held “Prototype 1” at the Face4 Systems Inc. office from July 10" to 12" 
2017. During the first two days, the project team ran four body-storming exercises for each of the pre- 
determined “Scenario Stories” in order to refine all the necessary data elements and determine the 
movement of these data elements between the various actors/systems. These exercises were low 
fidelity and paper-based in order to reflect our project principle of design thinking and rapid 
prototyping. Day three of Prototype 1 was reserved for discussing outcomes from the first two days, 
identifying the individual pieces of the system, determining the microteam leaders, determining the 
target scope for Prototype 2, and deciding how often the whole Design Team will be meeting. 


Going forward, the microteam leaders were tasked with assembling their teams, meeting with them to 
decide what they can (and can’t) deliver for Prototype 2 in September 2017, and building those pieces in 
time for Prototype 2 (whether real or simulated). 


It was also decided that the full project team will meet every two weeks (Thursday afternoons) for a 
Design Team meeting. The meetings will take place at the CBSA Lab (either the Colonnade or Bentley 
office) and project team members can attend in person or use the teleconference system to call in. The 
microteams will report on their progress at each of these meetings. The first Design Team meeting is set 
for July 25‘, 2017. 


This Prototype 1 report: 

The following document outlines some of the notes, questions, concerns, and comments from Prototype 
1 as recorded by Eric Galton (CBSA Lab) and (WorldReach). These can be considered as 
“Critical Project Drivers” as we continue through the rest of the project. 
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Scenario notes 
Below you will find notes that were written as the project team body-stormed the four scenario stories. 


Services frontaliers 


The notes from all four scenarios have been compiled and structured according to a rough timeline. 


Zone: Traveller Downloads, Sets-Up, and Uses the CBSA Mobile App: 


Mobile App (MyTravel) notes: 
e Need name and email address to start account. Some discussion about whether Passport # 
or other info required at this time. Traveller may be resistant to provide more. What would 

be incentive to do so? 


e App will have a CBSA chat session option, or simply an FAQ section which would allow for 
text search & retrieval of content to match query (FAQ should allow off line usage). 

e Category based content to help find information, will also allow off line usage. 

e Traveller should always have an indication of what events they still need to do (e.g. get an 
ETA, present at kiosk for payments, etc...). 

e App will allow traveller to request their travel history. CBSA will be able to find this 
information in passage records (CBSA Programs will address method of look-up). CBSA can 
then send list as PDF (would require an email out adding scope? this is not a commitment 
for the project, but a good stretch goal.) 

e Could be valuable to get voluntary provision/approval from traveller to share PNR data 
collected by reservation systems — to provide to CBSA. Incentive for traveller — expedited 
processing on arrival. If available provide to Orchestrator. 

e NOTE: CBSA system holds RBR not on the phone, will be in cloud site. 


Zone: Traveller ETA application: 


IRCC P.A. notes: 
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e [RCC will also pass Orchestrator the date of issue of ETA — important to know. 
e [RCC PA is never missing a score for an ETA. 


Zone: Traveller Check-In: 


Traveller Check-In notes: 
e At check in — additional info obtained from taking an OCR of passport biopage would benefit 
the traveller — ensures accurate data, no need to manually enter. 
e User responsibility to take this step — no prompt by system (assumed for now). 
e Later data may be pushed to traveller. 


VisionBox notes: 


e Info for VisionBox Orchestra — (from PRN, API, & from Orchestrator workflow) 
o Number of Canadians and non-Canadians on flight; 
o 
o Timing of flight arrivals (from Airport Authorities, other online sources). 
e CBSA Back office can use this info for operational purposes (assign BSOs to needed areas). 


Orchestrator workflow: 
e Once API data available, Orchestrator starts Trip Record. 
e Traveller mobile app will get updates from Orchestrator workflow — push/pull, integrated 
service bus, Other (means to be determined in micro groups). 


e Mobile app will interpret status from Orchestrator workflow & present message to the 
traveller. 


e Traveller state control — is separate from Orchestrator workflow — but is it handled by a 
different system? To be determined. 


CBSA Predictive Analytics notes: 
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2 themes that PA will fire on (travellers who are not the Criminal element): 


Will likely have 2 scores on 2 characteristics to feed to trip record — may be useful to keep 
these scores here, history of scores. 
Won't necessarily have all data — may be in situation where need more data from traveller. 


Zone: Traveller In Flight: 


Traveller in Flight notes: 


Requesting traveller to take RBR photo prior to landing may be an issue — if people try to 
take it on the plane it could be a distorted photo (taken too close); non-uniform lighting 
also can lead to poor quality. Will be some check on quality but may not be sufficient. Need 
to consider instructions for traveller & alternate time for photo to be done. Poor photo may 
require directing traveller to kiosk for second verification photo. This would be an event to 
Orchestrator -workflow. 

e-Declaration can be completed in transit (on airplane) but can’t be submitted until on 
Canada soil. App needs to ‘save’ data. 


Zone: Traveller De-plane: 


Traveller notes: 


E-Declaration barcode should only contain machine readable info from e-Declaration 
nothing more. 

If a traveller doesn’t have an RBR match once in primary will require going to Primary kiosk. 
No map or arrows in the Mobile App for now (we have removed all references to static or 
interactive maps/GPS from the scenario stories.) 


Orchestrator notes: 


Want to set a timer when plane lands to track time to Primary. 
Some question around what triggers this timer (indication from Airport Authority that plane 
down, first passengers start doing their Primary RBR confirmation, other?). 


What is passed to Mobile command — previous selfie or kiosk photo from passage record, 
other info from API (enough to identify the person). 
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Mobile command broadcast via radio to all officers to look for traveller, BSO who finds 
traveller takes this alert & actions it. 
As alert for traveller cleared by BSO (eg. Found traveller in arrival hall, no issue), 

for the traveller, the CBSA PA will be re-run for the traveller, 
Orchestrator reruns overall referral rules. 
Individual alerts have colour not cumulative. 
Overall referral is either 


Zone: Primary and Intermediate: 


Primary Kiosk notes: 


A lot of discussion on whether it is essential that the first time the CoT traveller goes 
through system must present at kiosk. Not taking advantage of ways that can authenticate 
traveller remotely in this approach. Needs further discussion. 

Looking for reasons why public will buy into this process — if have to use kiosk where is 
benefit? 


Kiosks need to be able to read the e-Passport. 


Facial recognition notes: 


Facial recognition prepares facial template from kiosk photo or selfie & photo at enrollment 
at their gate(?) 

Some question around how Facial recognition gets the kiosk photo or selfie from 
Orchestrator (push, pull, service bus). 

Each time a traveller enrolls at Facial recognition — event sent to Orchestrator & becomes 
part of trip record. 

If something has not allowed normal processing (e.g. person entered zone but didn’t come 
out), then facial recognition will require a reset of system. 

Reset will go to Orchestrator workflow so next event message can be sent. 


Orchestrator notes: 


Trip record will contain selfie taken at primary used to verify RBR. 


Mobile Command Dispatch System notes: 


Kiosks must be able to send message to Mobile Command (e.g. for failed RBR). 

Any dispatch pushed out to BSO from back office — e.g. alarm — to be on outlook for certain 
traveller. Dispatch may be to all BSOs, one who picks it up need to notify them have it, so 
task can be closed. If not picked up, back office can escalate. 
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e Communication by radio with back office won’t be going away — this dispatch supplements it 


doesn’t replace it. 


BSO_wrist-mounted mobile device notes: 

e Some questions about what information they will need access to... 

e BSOs should know what is pending for the traveller — what do they still need to do. 

e BSO should be able to take ownership of broadcast alert (once done, this will need to trigger 
an event in Orchestrator workflow to update the trip record). 

e BSO should be able to have either selfie from RBR verification or previous kiosk photo as 
well as other traveller data. 

e BSOs may want to have access to full list of incoming flight (# reds, greens) or should back 


office only have this? 


Time Delay scenario notes: 
e = =|f there is a time delay, the enforcement alert will trigger the re-run of the CBSA predictive 


analytics to gather the up-to-date information of the risk score. 

e To clear the time delay, questions will need to be asked by the BSO. If the answers satisfy 
the officer, the agent should also be able to check the risk score from the IRCC predictive 
analytics, in this present scenario, why the score is 


Zone: Traveller in Baggage Area: 


Missing Bag notes: 


Zone: Egress: 


Orchestrator notes: 
e Orchestrator never closes a trip. 
e Must allow for manual override of a status to become — based on behaviour etc. 


Facial Recognition notes: 
e = =If no enrollment of face has occurred prior to egress will allow enrollment at this point. This 


sends message to Orchestrator which confirms status 
e Facial recognition — once confirm traveller — will trigger closing of the trip record. 
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e = Facial recognition will not keep photos — clean up (every X amount of time). 


Services frontaliers 


Zone: Passport Renewal: 


Passport renewal notes: 

e in order to be able to notify traveller their passport is about to expire need to have passport 
info — therefore need to get passport info from them at some point (this is a good reason 
for asking for passport scan earlier than at check-in). 

e The traveller should be able to dismiss the notification in the Mobile App if they instead 
decide to pursue the renewal at a passport office. 

e When they get their new passport, traveller will have to update profile in My CBSA. 

e Mobile App will not notify traveller of automatic renewal, instead traveller will get “in 
process” notification. 


Concerns/questions that need to be addressed: 
Below are some important concerns and questions that were noted during Prototype 1. We will 


continue to address these issues as we proceed through the project lifecycle. 


1. Checked luggage vs. Carry Ons 
a. How to differentiate checked luggage from carry-on bags at Egress? (Some carry-on 
bags are quite large — How would the Egress security officer know which bags to count?) 


2. Is there a better way to get PNR information? 
a. Can we question the traveller on their mobile phone > OK to share? 
b. If yes, display what is to be shared. 
i. Validate/Approve 


3. Concern with RBR PIN 
a. Concern that travellers might not like it having to remember a PIN. 


5. What information does a BSO need to know to effectively do their job? 
a. What needs to go on the wrist-mounted device? 
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6. The difference between Primary and Intermediate kiosk zones 


Services frontaliers 


a. Based on the scenarios we ran, one seemed to focus on collecting identity and e- 
Declarations, and one for admin tasks. We are not sure if those are different from the 
point of view of the traveller. 

b. ls there a functional need to have the two zones? Can all flags be dealt with in the same 
zone? There could be less traveller confusion by simplifying this into one zone. 


7. Multiple passports 
a. How does IRCC/Passport Canada want us to handle the issue of multiple passports 
during the passport transition phase? (New passport has been approved and is valid but 
the traveller hasn’t picked it up yet.) What is their process? 


8. Events & Timers 

a. All events must be captured in the workflow. 

b. There was a discussion on the timer for primary. The Trip Assessment will have a built-in 
configuration file which will host an overall time it will take for a traveller to walk from 
the gate to the primary zone. The timer will be triggered by the first person from that 
flight who cleared primary. Is this still a viable concept? 


ltems that were decided during Prototype 1: 


Here below is a list of decisions/resolutions that were noted during the course of Prototype 1. 


1. System communication 
a. Systems communication will be based on: 
o Structure 
o Events. 


2. Common data elements: 
a. It has been identified that the following data elements are common during the traveller 

continuum: 
o Trip Record 
o Seed data to drive story 
o My Travel (+RBR) 
o GCMS 
o eTA 
o Passage Record. 
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3. Codes (programming) 
a. Note: Don’t focus on reusable codes, functions, classes, components. The focus is on the 
functionality, not efficiency, for Prototype 2. 


4. Overall risk score within the context of all risk scores 
a. It is understood that the overall risk score is an amalgamation of all the other risk 
scores. 
This is described in the example below: 


b. All global risk scores (IRCC P.A. and CBSA P.A.) are to be stored in the Trip Record and 
the evolution history is to be kept during the traveller’s continuum. 


5. CBSA “My Profile” 
a. The CBSA My Profile contains the following basic data elements: 
Name 
Email Address 
Passport number or OCR the passport page 
Photo (to be confirmed) 
Occupation (to be confirmed) 


oO 0 0 90 


Employer (to be confirmed) 


6. Primary Kiosk 
a. Inthe context of the prototype, the primary kiosk (which will share a lot of base code 
from the Mobile App), will be used to: 
o Take initial photo of 1* time Chain of Trust traveller (reliable photo conditions/ 
no distortions) 
Validate RBR codes 
Scan traveller’s e-passport 
Review e-Declaration (if needed) 
Use photo for enrolment into Face4 workflow system (?) 


OOF OO 


7. Orchestrator (Trip Assessment) 
a. The Orchestrator (Trip Assessment) will be used to: 
o Publish the state change 
o Publish the next state 
o Provide a subscription mechanism to the traveller and CBSA for current and 
future traveller’s state 
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b. The logic of the Trip Assessment should be able to be adjusted as per Port of Entry (POE) 
required staff and overall capacity. 


8. Wrist-Mounted Mobile Device 
a. The Egress and Secondary activities and actions will be managed via the BSOs wrist- 
mounted mobile device which will be capable of changing/updating a traveller’s Trip 
Record. 


9. Data Stores 
a. Two data stores that were missing in the original conceptual Technology Block Diagram 
have been identified and added to the overall architecture: 
CBSA “My Profile” 
CBSA “Mobile Command Dispatch System” (which will use a zone-based 
notification in the message envelope to target specific group of BSOs via their 
wrist-mounted device) 


10. The Trip Record 
a. The Trip Record will hold all information about the traveller during the continuum. It will 
be read and updated by the Trip Assessment engine, though other systems may also 
need access to some key data (Ex: Facial Flow Control for Egress). 
b. The Trip Records starts upon receipt of API data. 
11. IRCC Predictive Analytics 


a. The predictive analytics engine will need to expose, upon request, the following data 
elements: 
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b. These data elements will be stored in the Trip Record. 
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Things we need to do: 


1. Map of all events 
a. In order to achieve a perfect inter-system communication in the event’s flow, it is 
recommended to draft a map of all the events. 
2. RBR 
a. David will look at some suggestions/ideas for a “second hunch” generator (something 
that could change over time). 
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Annex Diagrams 
In the next four pages please find the following diagrams: 


o Annex 1: Chain of Trust Team Structure 

o Annex 2: Chain of Trust — Individual Pieces & their respective functionality (Microteam leaders 
listed in Red) 

o Annex 3: Chain of Trust — Architecture Diagram and Data Flow 

o Annex 4: Chain of Trust - Communications 


Page 12 of 16 
PROTECTION * SERVICE + INTEGRITY 


A-2018-09320-001126 


à Immigration, Refugees Immigration, Réfugiés 
and Citizenship Canada et Citoyenneté Canada 
s.1 9(1 ) Information disclosed under the Access to Information Act 


L'information divulguée en vertu de la loi sur l'accès à l'information 


Canada Border Agence des services 
Services Agency frontaliers du Canada 


ET O ET 


Annex 1: Chain of Trust Team Structure 


DESIGN TEAM 


Entrust: WorldReach: University of Ottawa: 
University of Ottawa: University of Ottawa: CBSA: Sopheap Ker 
CBSA: Colin Thompson | CBSA: Marc Leonard | CBSA: Eric Galton | 
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Chain of Trust (CoT) 
Exercise 2 - Outcomes and Notes 
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Date: October 10th, 2017 Reference: CoT20171011 


Exercise 2 Summary Report: 

The Chain of Trust project held its Exercise 2 at the CBSA Science and Engineering Facility from 
September 19° to 21% 2017. The exercise was designed to assess the system design by walking 
“travellers” through a simulated operational scenario. 


On Day 1 of the exercise, volunteer actors played the roles in a story (traveller, roving BSO, egress 
officer) and moved throughout our simulated “airport” while interacting with the various systems of the 
Chain of Trust project. These walk-throughs allowed the project partners (and micro-teams) to assess 
the functionality of their individual pieces, learn more about the system’s communication needs (risk 
assessment/information flow), and see the traveller flow in action. Overall technical progress on the 
project was more advanced than had been anticipated. Many of the communication links with the 
orchestrator were already operational, despite belief that most would require manual simulation at this 
stage. 


On Day 2 of the exercise, a smaller group of project partners worked to refine certain data 
communications elements based on findings from the previous day. On Day 3, the whole project team 
assembled to review the previous two days of work: identifying the elements that worked, the elements 
that did not, and fleshing out the “next steps” for each micro-team. The final portion of the day was 
devoted to planning for Exercise 3. 


Various additional observers from the CBSA and IRCC attended the prototype exercise to provide useful 
insight from their own domain expertise (such as a team from CBSA Programs). On the final day of the 


exercise, a small delegation from IRCC and Transport Canada were also provided with a walk-through of 
the early prototype and an overview of the project. 
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The Site 
The CBSA Science and Engineering Directorate has secured a project space at 79 Bentley Avenue in 


s.19(1) 


Services frontaliers 


Ottawa. This space will house the prototype exercises for the duration of the project. The project space 
has been configured to simulate an airport using both metal truss and stanchions to delineate various 
“zones” such as an Arrivals hall, Intermediate area, baggage carousel, and egress zone. Additional zones 
were also added to simulate the traveller’s “pre-departure period” (applying for eTA, buying tickets to 
Canada, in flight, etc.) as well as an area to simulate CBSA back office functions at the airport. 


Planning and Preliminaries 

In preparation for the Exercise 2, the Chain of Trust partners established micro-teams to advance 
specific thematic elements of the prototype. Each micro-team participated in weekly conference calls to 
discuss technical development, outstanding issues/challenges, and determine next steps. Larger ‘Design 
Team’ meetings were convened on a bi-weekly basis, which focussed on the overall progress, logistics, 
issues, and project management. Additionally, in the weeks leading up to the exercise, the project space 
at 79 Bentley was made available to the project partners for equipment setup and testing of their 
communication links with the orchestrator system. 


Exercise 2 

The Chain of Trust project team has developed several storylines (Annex B) in order to simulate various 
potential traveller/BSO/system experiences. Exercise 2 focussed exclusively on the simulation of 
Scenario #3 which showcases the continuous risk assessment of a foreign visitor arriving in Canada and 
proceeding through the travel continuum. This particular scenario was chosen because it would allow 
for assessment of all major sub-components of the project. Over the three day exercise, volunteer 
actors played the role of the traveller, moving from their initial travel planning (applying for an 
electronic travel authorization, check-in, etc.), all the way through to their egress from the Canadian 
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airport. This ‘scenario’ approach allowed for a practical assessment of both the system’s design and the 


user’s experience as they interacted with the technology. 


To protect privacy and sensitive information, no real passports or personal phones were used in Exercise 
2. Instead, the project team developed simulated traveller data, used specimen passports, and 
purchased mobile devices that will be used exclusively for Chain of Trust. Additionally, sensitive 
information relating to CBSA and IRCC operations was excluded from the scope of the exercise. 


Exercise 2 was the second of four exercises planned under the Chain of Trust Project, each progressively 
increasing in the level of technical fidelity. Exercise 1 was undertaken on July 10-12, 2017, and focussed 
on fleshing out the design and flow of the Chain of Trust project scenarios. Exercises 3 and 4 will 
examine further refinement on the progress established in exercise 2, finalizing remaining 
communication components, and expanding the design assessment to a larger number of operational 
scenarios. 


The Scenario #3 storyline is described below: 


Scenario #3 - Time delay triggered (International traveler) 

Marie Bennet is a French citizen who completed her undergraduate degree at the University of British Columbia on 
an approved student permit 10 years ago. Marie is now planning a trip from Paris, France, back to Vancouver, BC to 
visit some friends from university. Marie downloads the CBSA app on her mobile phone and creates her CBSA “My 
Profile” by following the instructions on the screen. She scans her passport, enters her personal information, and 
takes 7 selfie pictures of herself in order to generate her initial RBR code. As a foreign national from a visa-exempt 
country, Marie requires an eTA to visit Canada. She uses the app to complete her eTA application, pay the IRCC 
transaction fee, and send her application to IRCC for review. 


Marie’s eTA is approved within seconds and she receives notification 
of this approval on her mobile phone. 


Marie buys her tickets to Canada. Two days later, she arrives at Charles de Gaulle airport in Paris, France and checks- 
in for her flight. At this time the CBSA starts Marie’s “Trip Record” in the system. On the plane ride to Vancouver, 
Marie drafts her e-Declaration using the app on her mobile phone and confirms her declaration using her facial 
biometric (she takes a selfie on the plane). 


Continued next page... 
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Marie disembarks the plane in Vancouver and sees that she has wi-fi now available, so she opens her CBSA app 
and submits her e-Declaration. The app then notifies Marie that, as a new enrolee in the Chain of Trust program, 
she will need to present herself at a Primary kiosk in the Arrival hall for “photo validation.” 

Marie makes her way to the Arrival 
Hall. 


However, an hour passes by and the CBSA Risk Assessment system notes that “Marie Bennet” still hasn’t 
presented herself at a Primary kiosk, nor has the Face Recognition flow control system captured her moving 
anywhere past the Arrival hall. 


Andria Bingley, a CBSA Mobile Enforcement Officer, finds Marie Bennet sitting in the Arrival area. Andria 
using her wrist-mounted mobile device, then approaches Marie. It turns out that 
Marie’s mobile phone had been about to run out of power, so she had plugged her phone into a free outlet on 
the wall and was waiting for it to re-charge. After a lengthy interview, Andria determines that she sees no reason 
to keep Marie’s so she uses her wrist-mounted mobile device to downgrade Marie to 
in the system (since Marie still needs to complete her photo validation at a Primary kiosk). Andria then 
directs Marie to present herself at a Primary kiosk as soon as possible, and then goes back to her roving duties. 


Marie picks up her mobile phone and goes to kiosk #11 in the Primary zone. Kiosk #11 takes a photo of Marie 
and validates her RBR codes. Marie is now made n the system and directed to the baggage carousel in 
the Intermediate zone. Marie goes to the baggage carousel and easily picks up her bag. Marie is then directed 
by the app to enter one of the Egress lanes. In Egress lane #4, the Face Recognition flow control system 
recognizes “Marie Bennet” and shows that she is a craveller with “one” piece of luggage. Johnathon 
Darcy, the security officer at Egress lane #4, sees this information displayed on a screen and confirms that Marie 
does indeed have only one bag. Jonathan lets Marie proceed to the exit of the Egress zone and leave the airport. 
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Progress, Challenges, and Next Steps 


Services frontaliers 


To support the development of this report, each micro-team leader was asked to provide their notes 
from Exercise 2. These notes listed the elements that worked, those that did not, and outlined the 
remaining work that needs to be done. Summaries of these notes are provided below: 


> TRAVELLER MOBILE APP (“MY TRAVEL” APP): 
All steps required to execute scenario 3 on the mobile device were completed and available for the 
exercise. For the simulation, a ‘wizard’ interface was used to manually trigger the traveler’s next 
steps/available actions in the mobile app (RBR, eTA, e — Declaration). 


Next steps for the app will include: 


e = improving traveler notifications to make new messages more apparent to travelers, 

e an examination of some of the terminology used within the text of the app, 

e completion of the integration between the MyTravel App and the Orchestrator system, 
as well as between the MyTravel App and the Renewable Biometric Reference (RBR) 
Service. It is anticipated that this will be functional for Exercise 3. 

e = modification of the MyTravel app will also be made to allow for more than one trip to be 
sequentially initiated and completed. 

e Finally, next steps will include the design of the passport renewal component for 
Canadians using the MyTravel app. 


> KIOSKS 
Functionality of the ‘Primary’ kiosk was ready for Exercise 2 (including RBR validation and integration 
with a simulated RBR server). Furthermore, an enrollment function was developed in order to capture a 
photo of each actor and substitute it for the specimen passport photo in the MyTravel server. This 
allowed the project team simulate the scenario with real specimen passports. 


Feedback on the Primary kiosk’s processes indicated that the system was very simple and fast to use, 
though some usability elements could be addressed in future iterations (responsiveness of buttons, 
passport reader location, etc.). There was discussion of looking at acquiring other models of tablets 
which likely would not have NFC & therefore require use of an external NFC reader (NOTE: further 
research indicates that this will not be possible at this time due to a software issue — we have decided to 
forego this idea). 


Next steps for the kiosks will include: 
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e developing the ‘Intermediate’ kiosk for the administrative task of paying fees; 
e creating a check-in function to simulate what happens at airline check-in for beginning 
of the trip record. 


> ORCHESTRATOR 
By the end Exercise 2, many of the technology elements were communicating with the orchestrator 
system, though several elements of the prototype still required simulation through manual triggering 
(use of a “wizard”). 


Next steps for the Orchestrator will include: 


e align data fields, 
e incorporate traveler photos, 
e and add additional policies. 


It is anticipated that the system will be 100% rules-driven for Exercise #3 in December. 


> RENEWABLE BIOMETRIC REFERENCE (RBR) 
Next steps for the RBR elements include: 
e reformat some of the photos captured in the exercise, 
e load the RBR code to the WorldReach server. 
e The U of O team will also further examine the system’s robustness in dealing with 
variability of the photo quality, cameras used, etc. 


> IRCC PREDICTIVE ANALYTICS 


A realistic risk scoring system (built using simulated traveler data) developed under CSSP was 
demonstrated separately from the direct traveller flow using an IRCC laptop. This was done 
intentionally to avoid exposing the underlying code and scoring criteria. 
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Next steps for IRCC Predictive Analytics include: 


Services frontaliers 


e improve the risk scoring modeling to capture additional types of risk. 


> CBSA PREDICTIVE ANALYTICS 
CBSA Predictive Analytics was able to establish communications with the Orchestrator for the purposes 
of scoring. A client program was developed enabling the system to listen and parse the broadcast from 
the Orchestrator, query the server database for trip record, evaluate the risk score and send the result 
back to Orchestrator. 


Realistic data was simulated based on input from the CBSA and external sources such as Statistics 
Canada. Three distinct scoring models were built to assess risk of travelers with: (1) undeclared goods; 
(2) undeclared currency, and (3) intention to overstay. The models were designed to process the scoring 
at two time points in the traveller continuum — 


Next steps for CBSA Predictive Analytics include: 
e seeding the “backstory” for testing the CBSA Predictive Analysis Engine 
e moving the code from a laptop to a cloud environment 
e optimizing the communication with the Orchestrator 
e and continuing to tune and add complexity to the model to improve performance. 


> FACE RECOGNITION/TRAVELER FLOW 
Face4 computers and monitors were installed and connected to the CBSA-installed network and sensors. 
Face4 developed and installed face flow software and demonstrated the system using software buttons 
to simulate optical sensors. The focus was on demonstrating very fast capture, match, and display. In 
some cases, the camera was capturing faces of people outside of the traffic lane. There was some 
discussion about possibly using screens between egress lanes to prevent this. 


Next steps for Face Recognition/Traveler Flow will include: 


e integrating with the sensors and establishing communications with the Orchestrator. 
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e Additional operational input will also be collected to deal with exceptions, configure for 
multiple lanes/entrance lanes, examine usability/signage, and determine needs for the 
officer-facing display. 

e Face4 will continue to test in CBSA facility prior to Exercise 3. 


> CBSA MOBILE ENFORCEMENT APP AND DISPATCH SYSTEM 
For Exercise 2, the Mobile Dispatcher web service was able to establish communications with the 
Orchestrator queue, including querying for a traveler’s record. The system was able to send and receive 
messages to/from the Mobile Enforcement app via Firebase Cloud Messaging (FCM). 


Next steps for the CBSA Mobile Enforcement App and Dispatch System will include: 
e further development of the mobile enforcement scenarios, 
e improve the user interface, 
e and work out expected Traveller State messaging for the Orchestrator. 
e For the next prototype exercise, it is planned to have the mobile dispatch web service 
deployed to the server, have a basic functional version of the mobile dispatch and 
mobile enforcement app. 


OTHER FINDINGS (MISCELLANEOUS) 

e The process of taking multiple selfie and kiosk photos throughout the travel continuum (for 
identity verification) was perceived as rather onerous. Further discussions to come. 

e The use of mobile apps to undertake some activities (e.g., e-311 submission, passport scans) 
rests on an assumption of widespread wi-fi availability to travellers. 

e There was some discussion around potentially moving the face recognition to an earlier stage in 
the travel process. 

e There was a comment that it may be important to separate messages by zone or limit the 
number of messages seen by a particular BSO to only the most pressing. 

e Additionally, it was noted that the system could include a feedback mechanism for providing 
Border Services Officers with information on the outcomes of their referrals to secondary. 

e Volunteers felt that the mobile traveller app was very intuitive, with some minor suggestions for 
improvement (e.g., improved home screen navigation and more user-friendly terminology in the 
text). 
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NEXT STEPS: 


e Exercise #3 will occur in December 2017 and will focus on all remaining scenarios, including 


Services frontaliers 


two enforcement scenarios currently under development by the CBSA. 

e Project Partners will continue to undertake the regular bi-weekly Design Team meetings in 
the lead-up to the next prototype exercise. 

e = Micro-team leads and project partners were advised to have their list of next steps 
underway for the following Design Team meeting, scheduled for October 5". 


Annex Diagrams 
In the next four pages please find the following diagrams: 


o Annex 1: Chain of Trust — Individual Pieces & their respective functionality (Micro-team leaders 
listed in Red) 


o Annex 2: Chain of Trust — Exercise Scenarios 
o Annex 3: Chain of Trust —- Communications 
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ANNEX 1: Chain of Trust — Individual Pieces & their respective functionality (Microteam leader listed in 


Red) 


Orchestrator 


e Policies & details: 
e Linkage between 


Mobile Device 


(traveller) 


e My Travel (+RBR) 


IRCC Predictive Analytics 


e WR-eTA 
e GCMS 


BR 


eService core 
e Mobile device (My 


Analytics 


e Orchestrator events 

e Orchestrator data 

e Other data (passage 
records) 

e IRCC Predictive 
Analytics 


Microteam: 

U of Ottawa: 
CBSA Lab: David 
IRCC: Mike 


e £-Declaration 
e RBR 

e Photo 

e Passport read 
e Payment 


Microteam: 
CBSA BASD: Colin 
WorldReach: 
CBSA Lab: David 


e Events from 
Orchestrator 
> Enrollment 
> Indicate FR 
needed (required) 
> Indicate FR 
happened 
e Time response is a 
concern 
e Egress display 


Microteam: 
Face 4: 
CBSA Lab: David 


event and policies e eTA Future: Feedback from Travel) 
(mapping) e e-Declaration CoT e Kiosk 
e Events to publish 
e Passage data update Microteam: Microteam: 
Microteam: IRCC: Mike U of Ottawa: 
Microteam: WorildReach: WorldReach: CBSA Lab: 
Entrust: Identos hardware WorldReach 
CBSA Predictive Kiosk Primary Flow Control Mobile Enforcement 


App (MWMD) 


e Trip 

e Notification 
(dispatch) 

e Updates the trip 
record 


Microteam: 
CBSA BASD: Sopheap 


Kiosk Intermediate 
e Payment 


Microteam: 
CBSA BASD: Colin 
WorldReach: 
CBSA Lab: David 
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Annex 2 — Exercise Scenarios 
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Note that these scenarios are living documents subject to further refinement as the project progresses to 
ensure an operationally realistic scenario. Several additional scenarios, focussing on CBSA enforcement 
situations will also be developed for future exercises. *See Main text of this report for the description of 
exercise 3. 


Scenario #1 — Broken RBR (Canadian traveler) 


1. Canadian traveler drafts his e-Declaration on the airplane 
a. CBSA POE Administrator sees that extra flights will be arriving on the dashboard 
b. CBSA POE Administrator notifies CBSA Mobile Enforcement Command of extra flights 


2. Traveler arrives and Submits e-Declaration 

3. Overlimit 7 traveler directed to Intermediate 

4. Traveler tries to pay fees at intermediate kiosk, but the RBR fails — 

5. Kiosk notifies the Enforcement Officer 

6. Enforcement Officer goes to kiosk, talks to traveler, checks face, 
The STORY: 


Jim Dayne is a Canadian citizen who travels a lot and has re-entered Canada numerous times using the 
CBSA app on his mobile phone. He generated his initial RBR over 2 years ago, and therefore the CBSA 
system has established a level of trust in Jim and rarely requires him to validate his RBR at a Primary kiosk. 


Jim is returning home to Montreal, Quebec after a week-long vacation in El Paso, Texas. On the plane ride 
back to Montreal he drafts his e-Declaration using the app on his mobile phone and then confirms his 
declaration using his facial biometric. 


Daniel Tarth is the CBSA POE Administrator at Pierre Eliot Trudeau International airport. Early in the 
morning, Daniel looks at his operations dashboard (which provides information from the airport authority, 
iAPI, and other sources) and he sees that there are 20 incoming flights scheduled to arrive later in the 
afternoon (including Jim Dayne’s flight). Understanding that there will soon be an influx of travellers, 
Daniel tells Hannah Clegane at CBSA Mobile Enforcement Command of the extra incoming flights, 


Once the plane has arrived at the airport, Jim grabs his carry-on luggage, disembarks, and starts walking 
towards the Primary zone. While walking with the other travellers, Jim submits his e-Declaration on his 
mobile phone. The CBSA system compares Jim’s e-Declaration RBR with his original “My Profile” RBR and 
has no problem validating Jim’s e-Declaration identity. The CBSA app then notifies Jim that he will have to 
pay fees for the items he is bringing back to Canada at a kiosk in the Intermediate zone. 
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Jim goes to kiosk #13 in the Intermediate area to pay his fees. The kiosk asks Jim to present his passport, 
so he follows the on-screen instructions and the kiosk scans the MRZ chip on his passport. The kiosk then 
tells Jim to stand still for an RBR validation photo to confirm his identity. Jim smiles as the photo is taken 
and waits for further instruction. However, the CBSA system is unable to validate the RBR photo taken at 
the kiosk — it seems that the new RBR code is failing to match up with Jim’s “My Profile” RBR code. Three 
things happen at this time: (1) Jim (2) kiosk #13 notifies Jim that there is a 
problem with confirming his identity and tells him to stay at the kiosk while a customs officer comes to 
talk to him, and (3) the system sends an alert to a nearby CBSA Mobile Enforcement Officer requesting 
help at kiosk # 13. 


Brendan Martell, a CBSA Mobile Enforcement Officer stationed in the Intermediate zone, receives the 
notification from kiosk #13 on his wrist-mounted mobile device. Brendan takes ownership of the alert and 
then heads towards the kiosk. 


Brendan approaches Jim and explains why he is there. Brendan interviews Jim over the next few minutes 
and soon realizes that Jim is indeed who he says he is. Brendan decides that he sees no reason to maintain 
Jim’s threat level, so he downgrades Jim Dayne in the system. Brendan uses his mobile device 
to validate Jim’s face in the system and overrides the match error in the system. Brendan tells Jim he can 
now pays his fees. 


Jim pays his fees (and is now made in the system). 

Jim goes to the baggage carousel, retrieves his suitcase, and proceeds to the Egress zone. As he enters 

Egress lane #2, the Face Recognition flow control system identifies “Jim Dayne” and displays that he is a 
who should have “one” piece of luggage with him. 

Lydia Bronn, the security officer monitoring Egress lane #2, sees the displayed information and then looks 


over to confirm that Jim Dayne does indeed have only one bag. Lydia allows Jim to proceed forward, exit 
the Egress zone, and enter Canada. 


Scenario #2 — Missing bags (Canadian citizen) 
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1. Canadian traveler’s API indicates they checked 5 bags 
2. Traveler drafts his e-Declaration on the airplane and indicates 5 bags 
3. Traveler submits e-Declaration 
4. Traveler only sees 3 of his bags at baggage carousel 
5. Traveler reports 2 lost bags (fills e-A23 form) — gets receipt 
6. Traveler heads to Egress 
7. Face Recognition workflow links the traveler to number of bags 
8. Information displayed on screen for an officer for verification 
9. Send traveler to Mobile Enforcement Officer. 
10. Traveller shows A23 receipt 
11. OK 
The STORY: 


Nate Delacour is a Canadian citizen who is moving back to Canada after living in Dublin, Ireland for 2 years. 
Nate downloaded the CBSA app on his mobile phone, created his “My Profile,” and generated his initial 
RBR last year, but the CBSA system still sometimes requires him to check in at a Primary kiosk for RBR 
validation. When Nate checks in at the airport in Dublin, the airline sends his API to the CBSA Targeting 
centre and indicates that he has checked 5 pieces of luggage. 


On the plane ride home to Ottawa, Ontario, Nate fills in his e-Declaration using the app on his mobile 
phone, reports that he has 5 checked bags with him on the flight, and confirms his declaration using his 
facial biometric. On arrival at Ottawa International Airport, Nate grabs his carry-on luggage, disembarks, 
and walks towards the Primary zone with the other travellers. On the way, Nate submits his e-Declaration. 
The CBSA app then directs Nate to proceed to a Primary kiosk for RBR validation and generates a barcode 
on his mobile phone. 


Nate goes to kiosk #7 in the Arrival Hall. The kiosk scans the barcode on his phone and reads his e-passport. 
Kiosk #7 also takes an RBR validation photo of Nate and the CBSA system has no problem matching the 
RBR codes. The app then notifies Nate that he can proceed directly to the baggage carousel to pick up his 
luggage. 


Nate goes to pick up his bags at the international baggage carousel, but while all the other travellers on 
his flight have retrieved their luggage, he finds that only 3 of his bags have arrived. The CBSA app asks 
Nate if he has retrieved all his luggage — he answers “No.” Therefore the app directs him to go to the 
airport’s “Lost Baggage” counter. Frustrated, Nate talks to Henry Granger at the “Lost Baggage” counter. 
Henry helps Nate report his 2 lost bags by completing an e-A23 form on an airport kiosk. The airport kiosk 
prints out a paper A23 receipt for Nate to keep. Henry promises that the airport will contact Nate as soon 
as they figure out where his lost luggage has gone. The app now asks Nate if he has his paper A23 receipt 
— Nate answers “Yes.” The app now directs him to proceed to the Egress zone. 
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Nate enters Egress lane #4 and walks forward with his 3 bags on his push cart. As Nate walks past an 
overhead camera, the Face Recognition flow control system recognizes that he is a who 
should have “5 bags” with him 


Frank Potter, a security officer who is tasked with monitoring Egress lane #4, sees the information on the 
display screen, but then looks down and notices that Nate only actually has 3 bags with him. Frank gently 
waves Nate out of the Egress line, and calls over the CBSA Enforcement Officer stationed at Egress, Esther 
Diggory. 

Esther Diggory heads to Egress lane #4 for a quick conversation with Frank Potter about the reason why 
Nate was pulled aside. Esther then approaches Nate Delacour and asks him about his missing bags. Nate 
explains that he has already filed an e-A23 about his missing bags, and shows Esther his A23 receipt. Over 


the next minute, Esther concludes that she has no reason to believe that Nate is a threat. Esther then tells 
Nate that he can proceed to the exit of the CBSA processing area and leave the airport. 


*SEE MAIN TEXT OF REPORT FOR SCENARIO #3 


Scenario #4 — Passport Renewal (Canadian traveler 
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Canadian traveler gets mobile app for entering Canada 
Traveler creates “My Profile” (RBR) 
App indicates to the traveler that their passport is close to expire date. 
Traveler poses a question to CBSA on the mobile app 
CBSA answers the question (provides “Did You Know?” trivia) 
Allow trip history to be viewed (passport stamp) 
Traveler buys a ticket 
Check in (e-check in) 
Get passport info — flag passport expiry date to traveller 
. In flight 
a. Traveler fills e-Declaration 
b. Traveler buys duty free items 
c. Traveler updates e-Declaration 
11. Arrival 
a. Traveler submits e-Declaration 
b. App reminds traveler that passport is about to expire 
12. Primary Kiosk 
13. App invites traveler to start passport renewal process on mobile 


DE 00 ir PAUL OO S 


= 
O 


The STORY: 


Ruth Organa is a Canadian citizen who has been living with relatives in New Delhi, India for the past 3 
months and is about to travel back to Ottawa, Canada. While still in New Delhi, she downloads the CBSA 
app, creates her CBSA “My Profile,” generates her initial RBR, and scans her passport chip. The app 
indicates to Ruth that her Canadian passport is due to expire in 7 months. 


Ruth poses a question to the CBSA on the mobile app about duties and fees, and the app answers her 
question via a “Did You Know?” trivia section. 


Ruth also happens to be in the process of renewing her Top Secret clearance for the Government of 
Canada, and she knows that one of the questions on the clearance forms is about trip history over the 
past 10 years. So while she is still in the CBSA mobile app, she takes a few moments to request her 
“Passage History” list from the CBSA. 


The CBSA receives Ruth’s “Passage History” request. The list is drawn together, prepared into a PDF, and 
the document is emailed to Ruth within 2 days. 


Ruth buys her ticket back to Canada and checks in for her flight (e-check in). 


While on the flight, Ruth fills in her e-Declaration, reports that she has 1 checked bag with her, and 
confirms her declaration using her facial biometric. 2 hours later, on a whim, Ruth decides to buy some 
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duty-free bourbon on the flight, so she then updates her e-Declaration and once again confirms her 
declaration using her facial biometric. 


On arrival at the Ottawa International Airport, Ruth grabs her carry-on luggage, disembarks, and walks 
towards the Primary zone with the other travellers. On the way, Ruth submits her e-Declaration. The app 
tells Ruth that she is still under the limit for her declaration so she will not have to pay any duties or fees. 
However, as a first-time user of the Chain of Trust program, the app notifies Ruth that she will have to 
present herself at a Primary kiosk in the Arrival hall for “photo validation.” 
Ruth goes to Kiosk #5 in Arrival Hall, takes 
her RBR photo and the system has no problem matching her RBR codes. 
The app now notifies Ruth that she can therefore proceed directly to the baggage carousel in the 
Intermediate zone. 


Ruth picks up her luggage at the baggage carousel and then heads to the Egress zone. Ruth enters Egress 
lane #7 and walks past an overhead camera which is connected to the Face Recognition flow control 
system. As she walks forward, Luke Walker, the security officer stationed at Egress lane #7, 

Luke looks down 
and confirms that Ruth does indeed have only one bag, so he lets her proceed to the exit of the Egress 
zone and leave the airport. 


2 months later, and now that Ruth is back on Canadian soil, the app reminds Ruth that her passport is set 
to expire in 5 months and invites her to complete the passport renewal process on her mobile. Ruth 
appreciates the reminder and spends a few minutes completing her passport renewal application on the 
smartphone app. 


To verify her identity, Ruth is asked to scan her existing passport with her phone and to submit a digital 
picture of herself. The app provides instructions for how to take a photo of acceptable quality. Ruth 
completes her renewal application and submits her payment. Because Ruth’s photo is of acceptable 
quality, 
the app informs Ruth that her passport application has been received. 


The next day, Ruth receives an email notifying her that her renewal application has been approved, and 
provides her with the option to pick up the passport at her local passport office or have it mailed to her 
home. 
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Chain of Trust (CoT) 
Exercise 3 - Outcomes and Notes 
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Date: January 24, 2018 Reference: CoT20180124 


Exercise 3 Summary Report: 

The Chain of Trust project held its Exercise 3 at the CBSA Science and Engineering Facility from 
December 5° to 7", 2017 (with a pre-exercise testing session on site on December 4‘, 2017). The 
exercise was the third of four major exercises planned for Chain of Trust. Exercise 3 incorporated the 
lessons learned from the previous exercise held in September 2017. 


Exercise 3 followed the model of Exercise 2 by having volunteer actors play the role of “travellers” and 
“Roving BSOs” moving through the travel continuum as written in five pre-scripted operational scenarios 
(see Annex 2 and 3). Whereas Exercise 2 focussed on executing a single operational scenario, Exercise 3 
expanded the number of scenarios tested, including a new mobile enforcement scenario. The first four 
scenarios outlined the movements of individuals travelling to Canada, including simulated travel data for 
each step (from their initial trip planning all the way through to their eventual egress from the Canadian 
airport). A fifth scenario was used to showcase a potential “day-in-the-life” of two Roving BSOs who 
were dispatched with the mobile enforcement technology. This scenario-based approach allowed for a 
practical assessment of both the system’s design and the user’s experience as they interacted with the 
technology. 


As with the previous exercise, to protect privacy and sensitive information, no real passports or personal 
phones were used in the Exercise. Simulated traveller data was provided in the scenario scripts, and the 
travellers used specimen passports and dedicated mobile devices throughout testing. Additionally, 

sensitive information relating to CBSA and IRCC operations was excluded from the scope of the exercise. 


In addition to running the scripted scenarios, the project team spent some time “free-flow testing” the 
system — allowing the travellers to interact with the technology in an unscripted manner. The systems 
generally performed well through these tests, which were designed to give a broader sense of user 
experience, performance and robustness to more real-world scenarios. Notably, the testing allowed 
assessment of the robustness of the system to handling several differing types of travelers moving 
through the arrivals hall simulation at the same time. 
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In Exercise 2, some aspects of the system still required manual intervention to facilitate communication 


between project components and the system orchestrator. However, Exercise 3 demonstrated a higher 
degree of technological fidelity, with increased level of communications between systems and increased 
automation of the system as a whole. Exercise 3 also saw significantly improved communication 
between the Orchestrator system, the Mobile Enforcement Command, and Roving BSOs through the use 
of mobile and smart watch messaging. 


As with the previous exercise, guest observers from CBSA and IRCC were invited to attend a portion of 
the exercise to better understand the work and its connection to departmental objectives. Shortly after 
the exercise, the project partners also reconvened at the site to provide a demonstration of the system 
to a small delegation from McKinsey and Company, who the CBSA has engaged on a separate project. 


Exercise 3 — Final day observations: 
The final day of Exercise 3 was set aside for reviewing progress, noting challenges, and determining how 
to proceed going forward. 


Among the issues discussed: 

e The need for further validation to ensure that the orchestrator messages can be reliably sent 
and received in cases where messages are moving in higher volumes and short intervals. 

e The need for additional work to ensure the reliable functionality of the project’s Renewable 
Biometric Reference technology, which is used to securely validate traveler identity in a privacy- 
respecting manner. Strategies included incorporating additional training data to strengthen the 
matching algorithm. 

e Clarification of the project vocabulary: the project team spent some time standardizing some 
terms that had been causing communications problems between project partners due to 
differing interpretations by different micro-teams (e.g., “enrollment”, “traveler”, “zones”). 

e The possible need for a master orchestrator in addition to orchestration functions that are 
specific to the destination airport. 

e Also discussed was the flow control logic -- which is specific to the destination airport layout -- 
and where that logic would reside (orchestrator vs flow control system). 

e Changes to the information displayed to CBSA staff working at egress: An indicator for the 
number of bags a traveller is carrying will be replaced with one indicating the number of 
travelers within a group. For the project, this information is displayed largely to demonstrate 
that the system is capable of automatically displaying some useful information to the CBSA 
officer/staff working at the egress point. 

e Changes to the rhythm of work: 
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o Inthe lead up to Exercise 3, Chain of Trust partners had continued to work through 


micro-teams to advance system components, building on the information gathered from 
Exercise Two. Design Team meetings were held on a bi-weekly basis, and teams would 
undertake additional testing as required using the dedicated project space at 79 
Bentley. 

o At the end of Exercise 3 it was decided that the project team would change the rhythm 
of work going forward to Exercise 4. The project team will now meet weekly (Tuesday 
afternoons) for “Exploration” meetings to discuss work and divide tasks, while bi-weekly 
“Testing meetings” will be held on-site every second Thursday afternoon. 


Exercise 4, planned for March 6-9" 2018, will further develop the scenarios, address remaining 
challenges, and further refine the system’s operation and realism. 


Micro Team reports: Progress, Challenges, and Next Steps 

To support the development of this report, each micro-team leader was asked to provide their notes 
from Exercise 3. These notes listed the elements that worked, those that did not, and outlined the 
remaining work that needs to be done. Summaries of these notes are provided below: 


> TRAVELLER MOBILE APP (“MY TRAVEL” APP): 
As with Exercise 2, travellers were still able to complete their activities on the app (e.g. create 
profile, apply for an eTA if non-Canadian, complete and submit their e-Declaration, etc.); 
however with Exercise 3, we added new on screen message notifications. These messages 
popped-up on the traveller’s mobile screen and could tell them their next step in the process 
(eg. proceed to Primary, collect bags, proceed to egress, etc). This new function seemed to work 
very well, even when multiple travellers were completing tasks at same time. However, we did 
initially encounter some issues with the messages and functionality of the app when multiple 
travellers checked in for the same flight. 


Some issues with interaction from MyTravel app and the orchestrator were also uncovered as 
part of this exercise. For example, because the Canadian specimen passports used for the 
exercise all had the same name, date of birth and expiry, this created some confusion when 
trying to distinguish travelers during the scenario. This created some confusion when trying to 
identify which travellers were being flagged for lookout by Mobile Enforcement and for 
distinguishing travelers on the Orchestrator dashboard. This problem will be fixed by creating 
our own project-specific “passports” going forward. 
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After Exercise 2, WorldReach had coded the MyTravel app to only force the traveller to validate 
their RBR photo at a Primary kiosk on their initial trip into Canada, after which they could bypass 
this step. So in order to have the travellers “fly” multiple times and be sent to Primary on those 
subsequent trips, the trip records needed to be cleared each time the traveller went around. 
The plan had been to use all live interactions for Exercise 3, but this issue became a bit of a 
hassle as we continued with testing. Furthermore, RBR generation had to be artificially 
completed for a lot of the travellers using code WR had implemented for Exercise 2. 


Also noted, was that mobile enforcement was seeing some data from the WorldReach internal 
test environment (e.g. photo for a traveller — was showing as a tester from WorldReach instead 
of the Exercise participant who had created a profile with the passport). This situation was 
caused by the WorldReach server in both the test and Exercise environments having used the 
same passports and communicating with a single Orchestrator. There is need of separate 
environments. 


Next steps include: 

e Fix MyTravel and Orchestrator interaction to allow concurrent trips by multiple 
travellers. 

e Modify configuration of MyTravel app to not use NFC in the phone to read eChips in 
passports. This will allow the use of lots of paper based passports with different made 
up names for next exercise (resolve the issues caused by specimen passports having the 
data). 

e Create separate test and exercise environments in the Cloud for all partners to use. 


> KIOSKS (Check-in, Primary & Intermediate): 
The Check-In kiosks performed well with travellers able to complete the small number of steps 
as required for the check-in process. The Primary Kiosk worked well for most passport reading, 
although problems with verification of identity (via RBR) sometimes created issues for the 
kiosk’s operation. The Intermediate kiosk had no problem with paying fees and duties. 


Next steps include: 
e Modify check-in kiosk to allow identification of number of travellers in a group. 
e Modify Primary kiosk to not require reading of chip in passport (as noted for MyTravel 
app) 
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> CBSA MOBILE ENFORCEMENT APP AND DISPATCH SYSTEM 
The CBSA Mobile Enforcement aspects of the project worked well in isolation, although some 


Services frontaliers 


issues remain relating to system integration and the data movement between systems. Slow 
network speeds for the simulation also caused some issues. 


Next steps include: 
e Adda database to manage Mobile Enforcement Officers (MEOs) 
e Setup development and production environments 
e Fix significant bugs 
e Integrate beacons for the indoor traveler localization 
e Time permitting, integrate Google Glass for mobile officer display 


> RENEWABLE BIOMETRIC REFERENCE (RBR) 
The RBR service worked as it should without any technical or algorithmic problems; however the 
system had some trouble passing/failing specific travelers according to the needs of each 
scenario. This led to problems when trying to test particular scenarios. The RBR parameter was 
modified on the fly during the Exercise to various degrees of success for some scenarios, but not 
all conditions could be targeted simultaneously for all scenarios. 


The RBR back-end code and the web service part have been combined. The RBR service has 
been uploaded and is successfully running on the cloud and it worked as intended. It can 
automatically generate a client’s RBR, verify an RBR and generate and verify the e-declaration 
“biometric signature.” A photo format has also been agreed upon to match the training data 
with the selfies taken at different points during the process by WorldReach’s applications. 


The RBR parameter needs to be tuned to increase performance and accuracy. The RBR service 
accuracy issues are caused by the limited training and testing data available to us at this time. A 
better trained feature extractor will improve the results. Significant differences in lighting and 
camera quality between the enrollment photos and the kiosk validation pictures were 
responsible for some of the authentication failures seen during the three days of the exercise. 


Next steps include: 
e The data collected during the exercise will be looked at and analysed to select better 


parameters for the service. 
e Accuracy improvements will be researched. 
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e Explore the possibility of improving photo pre-processing: potentially with the 


assistance of Face4 
e Create a quantitative measurement for photo variance in enrollment photos (lower 
priority, but still valuable). 


> ORCHESTRATOR 
In general the orchestrator system functioned as expected. We were able to run through all the 
scenarios, the rules fired as expected, and the traveler flow was updated correctly. Messages 
were sent and received by appropriate systems. 


There were some issues relating to output messages sent too close together (virtually at the 
same time), which were missed by receiving systems. An artificial user-evoked delay was 
introduced to accommodate this in the scenarios. Timer automation, used in scenario three, 
was still performed manually. Record clearing between systems was not automated and often 
caused errors. For example, the travel record would be cleared, but the Face4 cache was still 
present causing facial mismatch. 


Artificial attempts to rectify real people with a passport that was not their own caused confusion 
with traveller coordination. We tried a stop-gap method in the orchestrator by renaming 
passport data to another traveller, but this was very hard to manage and in the end was 
abandoned. Traveller database records would get too big and slow down the system to an 
unworkable level. Picture data was too large and could cause network delays. 


Next steps include: 


e Remove picture data from the Travel Record to and set as attachment (instead of inline 
with the record). This should reduce database slowdowns on query. 

e Create a thumbnail and full size version of every picture received by Orchestrator to 
reduce network traffic of images. 

e Add the automation timer into orchestrator to allow time rules to be processed 
automatically. 

e Add the archive sequence and archive output message to allow other systems to 
synchronize cleanup. 

e Remove the artificial stop gap methods to synchronize travelers and passports. 
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> IRCC PREDICTIVE ANALYTICS 


For Exercise 2, a simple R script was used to demonstrate the real-time flow of the risk scoring 


within the eTA application process. The WorldReach server was used to simulate the real-time 
risk scoring as part of the eTA assessment process. A realistic machine learning risk scoring 
system (built using simulated traveler data) developed for this project was demonstrated 
separately from the direct traveller flow using an IRCC laptop. This was done intentionally to 
avoid exposing the underlying code and scoring criteria. This same configuration was used for 
the live exercise testing for Exercise 3 — that is, a simple script was used on the WorldReach back 
end to demonstrate the communication aspects of the real-time eTA scoring, with a more 
advanced realistic scoring model demonstrated separately on an IRCC laptop. 


For Exercise 3, IRCC Predictive Analytics further advanced the scoring algorithm used in Exercise 
2, including parallelizing the code for faster model training and the development of an 
interactive version of the risk scoring model, allowing users to query specific cases for 
examination. 

The intention was to replace the simple 
scoring script used in the live exercise with this model for Exercise 3. 


Next steps for IRCC Predictive Analytics include: 


e Provide the new modeling code for inclusion in the WorldReach back-end. 

e Completion of a risk dashboard designed to demonstrate how the IRCC officers might 
use this model for assessing organized fraud trends. 

e Exploration of the possibility of combining the high-accuracy “black box” and high- 
interpretability “white box” machine learning scoring models generated for this project. 


> CBSA PREDICTIVE ANALYTICS 
For Exercise 3, the ‘data suggested’ model has been pre-trained based on mock data, which has been 
constructed based on Border Service Officers’ heuristics and Statistics Canada data. The ‘data suggested’ 
model has been deployed on the cloud and can automatically update and send the corresponding scores 
to the Orchestrator system by listening to messages. The mock data is relatively biased, and the PA 
engine may not be a good indicator if the stream data doesn’t ‘look like’ the mock data. However, the 
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Predictive Analytics engine is good at learning patterns from the historical data itself. Some challenges 


have been faced relating to the time-sensitive heuristics, which usually are required for additional triggers 
and rules. An example of a time-sensitive heuristic derived from discussions with a Border Service Officer 
would be that the BSO notices that multiple passengers on a single flight are displaying an unusual cough, 
at the same time that a possible pandemic concerns have been in the media. In this scenario, all travelers 
arriving on this flight would be triggered to a in the orchestrator in order to quarantine them. 


Next steps include: 


e Developing a 'bad' travelers scenario/use case for which the predictive analytics engine can show 
a high-risk score with a decent explanation or guide. 
Additional discussions with Entrust relating to the possible development of a use case involving 


time-sensitive heuristics. 


> FACE RECOGNITION/TRAVELER FLOW 
In general, the flow control worked as expected and no major problems were observed. All face 
images submitted to the central database were successfully enrolled into local databases for all 
lanes. Upon receiving an Egress message all traveler data was removed from local databases for 
all lanes. Lanes with 4 and 3 sensors performed equally well. 


Next steps include: 


e Switching to a single database for all lanes. This will help with synchronizing the 
watchlist for all lanes. 

e Incase of an exception, (e.g., someone walks through a lane in the opposite direction), 
the lane controller requires a manual reset. We will implement an automatic reset after 
a certain period of time. 

e Continue to fine-tune the state machine. 

e Implement changes related to integration with other components. The following 
expected changes will affect the work of flow control: Changes to the 
Development/Staging environment, Orchestrator, Central Database etc. 
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Annex documents 
In the following pages please find the following documents: 


o Annex 1: Scenario stories used in Exercise 3 
o Annex 2: Scenario scripts for participants used in Exercise 3 
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Annex 1 — Scenarios stories used for Exercise 3 
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Note that these scenarios are living documents subject to further refinement as the project progresses to 
ensure an operationally realistic scenario. Several changes will be developed for future exercises. *See 
Main text of this report for the description of how these stories were used in Exercise 3. 


SCENARIO STORIES 


Scenario #1 — Broken RBR (Canadian traveller) 


Jim Dayne is a Canadian citizen who is returning home to Montreal, Quebec after a week-long vacation 
in El Paso, Texas. On the plane ride back to Montreal he drafts his e-Declaration using the MyTravel app 
on his mobile phone (he reports that he is bringing 4 bottles of expensive bourbon back with him) and 
then ‘signs’ his declaration using his facial biometric. 


Daniel Tarth is the CBSA POE Administrator at Pierre Eliot Trudeau International airport. Early in the 
morning, Daniel looks at his operations dashboard (which provides information from the airport 
authority, iAPI, and other sources) and he sees that there are 20 incoming flights scheduled to arrive 
later in the afternoon (including Jim Dayne’s flight). Understanding that there will soon be an influx of 
travellers, Daniel tells Hannah Clegane at CBSA Mobile Enforcement Command of the extra incoming 
flights, and she quickly calls in 5 extra Enforcement Officers to work during those hours. Hannah can 
decide where to assign these officers based on traveller flow needs. 


(Check-In) (Flight Departure) (Draft e-Declaration) (Flight Arrival) 


Once the plane has arrived at the airport, Jim grabs his carry-on luggage, disembarks, and starts walking 
towards the Primary zone. While walking with the other travellers, Jim submits his e-Declaration on his 
mobile phone. The CBSA system compares Jim’s e-Declaration RBR with the RBR from his MyTravel 
profile and has no problem validating Jim’s e-Declaration identity. However, the MyTravel app on Jim’s 
mobile phone tells him that he has two administrative tasks to complete today: (1) the CBSA system has 
decided to validate Jim’s RBR today, so he will have to go to a Primary kiosk for an RBR validation photo, 
and (2) he has exceeded his alcohol allowance and will have to pay his duties and taxes at a kiosk. | 


Jim goes to kiosk #13 in the Primary area. The kiosk asks Jim to present his passport, so he follows the 
on-screen instructions and the kiosk scans the MRZ chip on his passport. The kiosk then tells Jim to 
stand still for an RBR validation photo to confirm his identity. Jim smiles as the photo is taken and waits 
for further instruction. However, the CBSA system is unable to validate the RBR photo taken at the kiosk 
— it seems that the new RBR code is now failing to match up with Jim’s MyTravel RBR code. Three things 
happen at this time: (1) Jim is i (2) kiosk #13 notifies Jim that there is a 
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problem with confirming his identity and tells him to stay at the kiosk while a customs officer comes to 
talk to him, and (3) the kiosk sends an alert to the Orchestrator about the RBR issue. 


While Jim is waiting at the kiosk, the Orchestrator forwards the alert to the CBSA Mobile Enforcement 
Command system. Hannah Clegane at Mobile Enforcement Command sees the alert and dispatches BSO 
Brendan Martell, a CBSA Roving Officer currently stationed in the Primary zone, to confirm the identity 
of the traveller at kiosk #13. 


BSO Brendan receives the notification on his wrist-mounted mobile device, acknowledges the alert, and 
then heads towards kiosk #13. 


BSO Brendan approaches Jim and explains why he is there. BSO Brendan interviews Jim over the next 
few minutes and soon realizes that Jim is indeed who he says he is. BSO Brendan decides to override 
Jim’s RBR match error in the system - he uses his mobile device to override the RBR error. Jim 
Dayne now turns back to the kiosk to continue with paying his duties and taxes. 


Jim pays his fees (and is now made in the system). 


Jim goes to the baggage carousel, retrieves his suitcase, and proceeds to the Egress zone. As he enters 
Egress lane #2, the Face Recognition flow control system identifies “Jim Dayne” and displays that he is a 
who should have “one” piece of luggage with him. 


Lydia Bronn, the security officer monitoring Egress lane #2, sees the displayed information and then 
looks over to confirm that Jim Dayne does indeed have only one bag. Lydia allows Jim to proceed 
forward, exit the Egress zone, and enter Canada. 


Scenario #2 — Missing bags (International traveller) 


Nate Delacour is an Irish citizen who is going to visit Canada from Dublin, Ireland. Nate has travelled to 
Canada 6 times in the last 2 years. He has already downloaded the MyTravel app on his mobile phone, 
created his Profile, and generated his initial RBR last year, and received his eTA. However, the CBSA 
system still sometimes requires him to check in at a Primary kiosk for RBR validation. When Nate checks 
in at the airport in Dublin, the airline sends his API to the CBSA Targeting centre and indicates that he 
has checked 4 pieces of luggage. 


(Check-In) (Flight Departure) 


On the plane ride home to Ottawa, Ontario, Nate fills in his e-Declaration using the app on his mobile 
phone, and ‘signs’ his declaration using his facial biometric. 
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(Draft e-Declaration) (Flight Arrival) 


On arrival at Ottawa International Airport, Nate grabs his carry-on luggage, disembarks, and walks 
towards the Primary zone with the other travellers. On the way, Nate submits his e-Declaration. The 
MyTravel app then directs Nate to proceed to a Primary kiosk for RBR validation. 


Nate goes to kiosk #7 in the Arrival Hall. Nate presents his e-passport to the kiosk and then stands still 
for an RBR validation photo. The system quickly makes a biometric match to Nate’s MyTravel RBR code, 
so the MyTravel app notifies Nate that he can proceed directly to the baggage carousel to pick up his 
luggage and proceed to the Egress zone. 


Nate goes to pick up his bags at the international baggage carousel, but while all the other travellers on 
his flight have retrieved their luggage, he finds that only 2 of his bags have arrived. Frustrated, Nate 
walks over to the “Lost Baggage” counter and talks to Henry Granger. Henry helps Nate report his 2 lost 
bags by completing an e-A23 form on an airport kiosk. The airport kiosk prints out a paper A23 receipt 
for Nate to keep. Henry promises that the airport will contact Nate as soon as they figure out where his 
lost luggage has gone. The MyTravel app now directs him to proceed to the Egress zone. 


Nate enters Egress lane #4 and walks forward with his 2 bags on his push cart. As Nate walks past an 
overhead camera, the Face Recognition flow control system recognizes that he is a ‘ 
who should have “4 bags” with him. 


Frank Potter, a security officer who is tasked with monitoring Egress lane #4, sees the information on 
the display screen, but then looks down and notices that Nate only has 2 bags with him. Frank gently 
waves Nate out of the Egress line, and calls over the CBSA Enforcement Officer stationed at Egress, 
Esther Diggory. 


Esther Diggory heads to Egress lane #4 for a quick conversation with Frank Potter about the reason why 
Nate was pulled aside. Esther then approaches Nate Delacour and asks him about his missing bags. Nate 
explains that he has already filed an e-A23 about his missing bags, and shows Esther his A23 receipt. 
Over the next minute, Esther concludes that she has no reason to believe that Nate is a threat. Esther 
then tells Nate that he can proceed to the exit of the CBSA processing area and leave the airport. 


Scenario #3 — Time delay triggered (International traveller) 
Marie Bennet is a French citizen who completed her undergraduate degree at the University of British 


Columbia on an approved student permit 10 years ago. Marie is now planning a trip from Paris, France, 
back to Vancouver, BC to visit some friends from university. Marie downloads the MyTravel app onto 
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her mobile phone and creates her Profile by following the instructions on the screen. She scans her 
passport, enters her personal information, and takes 7 selfie pictures of herself in order to generate her 
initial RBR code. As a foreign national from a visa-exempt country, Marie requires an eTA to visit Canada. 
She uses the app to complete her eTA application, pay the IRCC transaction fee, and send her 
application to IRCC for review. 


Marie’s eTA is 
approved within seconds and she receives notification ot this approval on her mobile phone. 


Marie buys her tickets to Canada. Two days later, she arrives at Charles de Gaulle airport in Paris, France 
and checks-in for her flight. At this time the CBSA starts Marie’s “Trip Record” in the system. 


(Check-In) (Flight Departure) 


On the plane ride to Vancouver, Marie drafts her e-Declaration using the app on her mobile phone and 
‘signs’ her declaration using her facial biometric (she takes a selfie on the plane). 


(Draft e-Declaration) (Flight Arrival) 


Marie disembarks the plane in Vancouver and sees that she has wi-fi now available, so she opens her 
MyTravel app and submits her e-Declaration. The app then notifies Marie that, as a new enrolee in the 
Chain of Trust program, she will need to present herself at a Primary kiosk in the Arrival hall for “photo 
validation.” 

` Marie makes her way to the Arrival Hall. 


However, an hour passes by and the Orchestrator system notes that “Marie Bennet” still hasn’t 
presented herself at a Primary kiosk, nor has the Face Recognition flow control system captured her 
moving anywhere past the Arrival hall. 


Andria Bingley, a CBSA Mobile Enforcement Officer, finds Marie Bennet sitting in the Arrival area. Andria 
using her wrist-mounted mobile device, then approaches Marie. It 
turns out that Marie’s mobile phone had been about to run out of power, so she had plugged her phone 
into a free outlet on the wall and was waiting for it to re-charge. After a lengthy interview, Andria 
determines that she sees no reason to keep Marie’s so she uses her wrist-mounted 
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mobile device to downgrade Marie to in the system (since Marie still needs to complete her 
photo validation at a Primary kiosk). Andria then directs Marie to present herself at a Primary kiosk as 
soon as possible, and then goes back to her roving duties. 


Marie picks up her mobile phone and goes to kiosk #11 in the Primary zone. Kiosk #11 takes a photo of 
Marie and validates her RBR codes. Marie is now made “ in the system and directed to the 
baggage carousel in the Intermediate zone. Marie goes to the baggage carousel and easily picks up her 2 
bags. Marie is then directed by the app to enter one of the Egress lanes. 


In Egress lane #4, the Face Recognition flow control system recognizes “Marie Bennet” and shows that 
She is a “ ’ traveller with “one” piece of luggage. Johnathon Darcy, the security officer at Egress 
lane #4, sees this information displayed on a screen and confirms that Marie does indeed have only one 
bag. Jonathan lets Marie proceed to the exit of the Egress zone and leave the airport. 


Scenario #4 — Passport Renewal (Canadian traveller) 


Ruth Organa is a Canadian citizen who has been living with relatives in New Delhi, India for the past 3 
months and is about to travel back to Ottawa, Canada. While still in New Delhi, she downloads the 
MyTravel app, creates her Profile, generates her initial RBR, and scans her passport chip. The app 
indicates to Ruth that her Canadian passport is due to expire in 7 months. 


Ruth has a question about duties and fees, so she taps the “Frequently Asked Questions” tab on the 
mobile app and navigates through her specific questions. The app is very helpful and answers all her 
questions. 


Ruth also happens to be in the process of renewing her Top Secret clearance for the Government of 
Canada, and she knows that one of the questions on the clearance forms is about her trip history over 
the past 10 years. So while she is still in the MyTravel mobile app, she takes a few moments to request 
her Passage History from the CBSA. 


The CBSA receives Ruth’s Passage History request. The list is drawn together, prepared into a PDF, and 
the document is emailed to Ruth within 2 days. 


Ruth buys her ticket back to Canada and checks in for her flight. 
(Check-In) (Flight Departure) (Draft e-Declaration) 
While on the flight, Ruth fills in her e-Declaration and ‘signs’ her declaration using her facial biometric. 


Two hours later, Ruth realizes that she had accidently put the wrong flight number on her declaration, 
so she re-opens her e-Declaration and easily changes this information. 
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(Flight Arrival) 


On arrival at the Ottawa International Airport, Ruth grabs her carry-on luggage, disembarks, and walks 
towards the Primary zone with the other travellers. On the way, Ruth submits her e-Declaration. Since 
Ruth is still under the limit for her declaration, she will not have to pay any duties or fees; however, as a 
first-time user of the Chain of Trust program, the app notifies Ruth that she will have to present herself 
at a Primary kiosk in the Arrival hall for “photo validation.” 


Ruth goes to Kiosk #5 in Arrival Hall, takes her RBR photo and the system has no problem matching her 
RBR codes. The MyTravel app now notifies Ruth that she can 
therefore proceed directly to the baggage carousel in the Intermediate zone. 


Ruth picks up her luggage at the baggage carousel and then heads to the Egress zone. Ruth enters Egress 
lane #7 and walks past an overhead camera which is connected to the Face Recognition flow control 
system. As she walks forward, Luke Walker, the security officer stationed at Egress lane #7, 

traveller who should have one piece of luggage. Luke looks down 
and confirms that Ruth does indeed have only one bag, so he lets her proceed to the exit of the Egress 
zone and leave the airport. 


Two months later, and now that Ruth is back on Canadian soil, the app reminds Ruth that her passport is 
set to expire in 5 months and invites her to complete the passport renewal process on her mobile. Ruth 
appreciates the reminder and spends a few minutes completing her passport renewal application on the 
smartphone app. 


To verify her identity, Ruth is asked to answer some standard passport renewal questions and submit a 
new digital picture of herself (the app provides instructions for how to take a photo of acceptable 
quality). Once finished filling in the information and taking her photo, Ruth submits her renewal 
application and pays the renewal fee with her credit card. She is presented with a QR code containing 
her pertinent data, that can be presented if she needs to visit the passport office to review her 
application. 


Soon after, Ruth receives an email notifying her that her renewal application has been received by IRCC 
and is now being processed. IRCC reviews Ruth’s updated information and photo, and does all their 
usual background checks on “Ruth Organa.” 


4 days later, the mobile app alerts Ruth that her application has been approved and alerts her that she 
can pick up her new passport at her local passport office. 
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Scenario #5 — BSO Scenario (A day in the life of a Roving BSO): 
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> BSO Devin Kent starts his day at the Hatfield International Airport. He reports to his supervisor 
and is informed that he will be on Roving duties today. 

> BSO Devin reports to the Mobile Enforcement Command office and talks to BSO Thérèse Wayne 
who is working as the dispatch officer today. 

> BSO Thérèse gives BSO Devin one of the mobile devices and a watch. 

> BSO Devin takes the mobile device and signs into the Mobile Enforcement Command system 
using his Username and Password. 

> Once the Mobile Enforcement Command system has ‘Activated’ ‘BSO Devin’ and located his 
mobile device with geo-location, the system sends a notification to BSO Devin’s mobile device 
which tells him that he can proceed with his Roving duties. 

> BSO Thérèse now uses the Mobile Enforcement Command system to dispatch BSO Devin to the 
baggage carousel area. 

> BSO Devin taps the “Roving” button on the watch screen and then heads out to the baggage 
carousel area. 

> On arrival, the system notifies Mobile Enforcement Command that BSO Devin has arrived at his 
assigned zone (using geo-location). 

> After a bit of roving, BSO Devin sees a traveller behaving suspiciously and decides to talk to him. 

> BSO Devin taps the "Engaged" button on the watch screen and walks up to the traveller. 

> BSO Devin asks the traveller for their name — “Jack Napier” — and then his passport. BSO Devin 
scans the traveller's document with his mobile device. 

> BSO Devin now sees Jack Napier's electronic profile on his mobile device and reviews the 
information that is on screen. 

> BSO Devin conducts a quick interview with the traveller, and soon suspects that Jack Napier may 
be bringing illegal substances into Canada. 

> BSO Devin smiles, says goodbye, and walks away from Mr. Napier. 


> BSO Devin also marks his suspected reasons for the referral by picking from a subsequent drop- 
down list: 
L Drugs, weapons, target, goods, immigration, other. 


(Jack Napier will now be diverted at Egress) 

BSO Devin now taps the "Roving" button on his watch screen. 

An hour later, BSO Devin receives a notification from the Mobile Enforcement Command that he 

has been re-assigned to the Arrival Hall. 

> BSO Devin taps the “Acknowledged” button on his screen and starts walking towards the Arrival 
Hall (the geo-location system knows where he is as proceeds through the airport). 

> On his way there, BSO Devin receives a Red Alert for reinforcement. His mobile device tells him 

to look for Officer Sylvie Prince who is at a location 100ft away from him in the Intermediate Hall 

(his mobile device shows her approximate location on screen through geo-location). 
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> BSO Devin taps the "Responding" button and starts running through hundreds of 
Travellers towards the location shown on screen. (By tapping the “Responding” button, this also 
marks BSO Devin as "Engaged" in the system without him having to press another button). 

> BSO Devin finds Officer Sylvie and helps her subdue a drunk and unruly traveller in the 
Intermediate Hall. 

> Once finished, BSO Devin taps the "Roving" button again, and continues towards the Arrival Hall 
(the geo-location system knows where he is as moves). 

> Onarrival, the system notifies Mobile Enforcement Command that BSO Devin has arrived to his 
re-assigned zone. 

> A moment later, BSO Devin receives notification from the Mobile Enforcement Command that 
there is a broken RBR issue at Kiosk # 6 - "Please confirm identity of traveller” 

> BSO Devin approaches the traveller at Kiosk # 6, taps his “Engaged” button, and talks to the 
traveller — Veronica Pennyworth. 

> BSO Devin quickly confirms Veronica’s identity and downgrades her from in the system. 

> He overrides the RBR issue using his mobile device. This override tells the system to continue 
with the rest of Veronica Pennyworth’s administrative processes. 

> BSO Devin then says goodbye to Veronica and taps the “Roving” button again. 

> An hour later, BSO Thérèse receives an alert from the Orchestrator that the CBSA database has 
just flagged an individual as a national security “lookout.” The Orchestrator sees that this 
individual, Peter E. Nigma, has just arrived at Hatfield airport. 

> BSO Thérèse now takes this information and sends it out to all active Roving BSOs. 

> BSO Devin receives the lookout alert for a traveller that should be somewhere in the Arrival 
Hall. 

> BSO Devin looks at his mobile device to review the known information about the traveller 
(Name: Peter E. Nigma, Sex: Male, Age: 42, Ethnicity: White, Hair colour: Black, Last seen 
wearing: Green jacket and jeans) — along with any available pictures (passport, profile photo, 
scene photo, etc.) 

> BSO Devin keeps roving and soon comes across the traveller in question. 


Services frontaliers 


> BSO Devin takes ownership of the lookout alert (which cancels it for everyone else). 
> BSO Devin now taps the "Engaged" button, and approaches the traveller to conduct an 
interview. 
Etc. Etc. Etc. 
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Annex 2: Scenario Scripts for Participants 


Scenario # 1 


Jim Dayne Hannah Clegane 
(Canadian traveller) (Mobile Enforcement Command) 
Step 1: Profile Creation 


Jim Dayne opens the MyTravel app and 
creates his profile: 


User Email address 

PIN: 6666 

Scan the picture page of your 
passport (make sure the bottom 2 
lines of the page are in the scan 
window of the camera) 

Take 1 Profile photo (your 
normal, relaxed face} 

Take 7 RBR photos (various facial 
gestures) 


Jim is ready to fly! 
Step 1: Assign Roving BSO to a Zone 


Hannah assigns Brendan to the Primary zone 
of the airport. 


Step 2: Check In 
Jim Dayne “Checks In” at the airport: 


Scan the picture page of your 
passport. 

Flight: AC689 

Bags: 1 


* PRESS THE “FLIGHT DEPARTURE” BUTTON 
FOR FLIGHT AC689. 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
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BSO Brendan Martell 
(Roving BSO) 
Step 1: Sign In to MEC system 


BSO Brendan Martell signs into the Mobile 
Enforcement Command system. 


7 


Step 2: Tap “Roving” button on watch 


Step 3: Walk to Primary 


Step 4: Check In to Primary 


On arrival, BSO Brendan “checks in” to the 
Primary zone using the mobile device. 
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Step 3: Prepare e-Dec 


FAR Services frontaliers 


In flight, Jim Dayne prepares his e- 
Declaration: 


Exemptions: $4000 
Flight: AC689 
Take RBR Photo 


* PRESS THE “FLIGHT ARRIVAL” BUTTON 
FOR FLIGHT AC639. 


Step 4: Submit e-Dec 


Jim Dayne walks off the plane and can now 
press the “Submit” button for his e- 
Declaration. 


The MyTravel app notifies Jim that he has to 
go to Primary for (1) photo validation and (2) 
to pay his duties and fees. 


Step 5: Primary Kiosk 


Jim Dayne goes to kiosk #13 in the Primary 
zone. At the kiosk: 


Scan the picture page of your 
passport 

Scan passport chip. L Close 
your passport and hold it next to 
the back of the kiosk (keep it still 
for 10 seconds as the reader tries 
to find the chip) 

Take RBR photo 


*THERE IS A PROBLEM WITH THE RBR 
PHOTO* 


The kiosk tells Jim to stay at the kiosk while 
someone comes to help him. 


+ 


Step 2: See Alert 
The Orchestrator notifies Mobile 
Enforcement Command that there is a 


problem verifying a traveller at kiosk # 13 in 
the Primary zone. 
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Jim Dayne hands his passport to BSO 


Brendan. 


Step 6: Pay Duties and Fees 


Jim Dayne now continues to the next step on 
the kiosk — paying his duties and fees. 


Complete the fields on screen. 
Credit card: 4111111111111111 


The MyTravel app notifies Jim that he can 


proceed to the Baggage Carousel zone to pick 


up his suitcase. 


Agence des services 
fronialiers du Canada 


Step 3: Forward the Alert to Roving BSO 


Hannah Clegane forwards the alert to 
Brendan Martell. 
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Step 5: See the Alert 


BSO Brendan Martell receives the alert and 
walks to kiosk # 13. 


Step 6: BSO Brendan presses the “Engaged” 
button on his watch. 


Step 7: Scan traveller’s passport 
BSO Brendan talks to Jim Dayne. 


BSO Bredan asks for Jim’s passport and scans 
it with his mobile device. 


BSO Brendan now reviews Jim Dayne’s travel 
profile and information on his mobile device. 


Step 8: Override RBR issue 


BSO Brendan overrides the RBR problem 
using his mobile device. 


Step 9: BSO Brendan presses the “Roving” 
button on his watch. 
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Step 7: Baggage Carousel 


Jim Dayne picks up his bag and proceeds to 
the Egress zone. 


Step 8: Egress 


Jim Dayne enters Egress Lane #2. 


s.16(1)(c) 


Agence des services 
frontaliers du Canada 


Lydia Bronn 
(Egress Security Officer) 
Lydia Bronn sees that “Jim Dayne” is a 


who should have “1” piece 
of luggage. 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


FAR Services frontaliers 


Lydia lets Jim proceed. 


Jim enters Canada. 


SCENARIO #2 


Nate Delacour 
{International traveller) 
Step 1: Profile Creation 


Nate opens the MyTravel app and creates his 
profile: 
User Email address 
PIN: 6666 
Scan the picture page of your 
passport (make sure the bottom 2 
lines of the page are in the scan 
window of the camera) 
Take 1 Profile photo (your 
normal, relaxed face) 
Take 7 RBR photos (various facial 
gestures) 
Apply for eTA 
Pay with your credit card: 
4111111111111111 


Nate is ready to fly! 


Henry Granger 


(Lost Baggage Counter) 


Frank Potter 


(Egress Security Officer) 
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Step 2: Check In 


FAR Services frontaliers 


Nate Delacour “Checks In” at the airport: 


Scan the picture page of your 
passport. 

Flight: AC425 

Bags: 4 


* PRESS THE “FLIGHT DEPARTURE” BUTTON 
FOR FLIGHT AC425. 


Step 3: Prepare e-Dec 


In flight, Nate Delacour prepares his e- 
Declaration: 


Allowance: $0 
Flight: AC425 
Take RBR Photo 


* PRESS THE “FLIGHT ARRIVAL” BUTTON 
FOR FLIGHT AC425. 


Step 4: Submit e-Dec 


Nate Delacour walks off the plane and can 
now press the “Submit” button on his e- 
Declaration. 


The MyTravel app notifies Nate that he has 
to go to Primary for photo validation today. 


Step 5: Primary Kiosk 


Jim Dayne goes to kiosk #13 in the Primary 
zone. At the kiosk: 


Scan the picture page of your 
passport 

Scan passport chip. L Close 
your passport and hold it next to 
the back of the kiosk (keep it still 
for 10 seconds as the reader tries 
to find the chip) 

Take RBR photo 


The MyTravel app notifies Nate that he can 


proceed to the Baggage Carousel zone to pick 
up his suitcase. 
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Step 7: Baggage Carousel 


FAR Services frontaliers 


Nate Delacour picks up 2 of his 4 bags, but 
can’t find the other 2. 


Nate goes to the Lost Baggage counter. 
Step 1: Lost Baggage Counter 
Henry Granger shows Nate how to fill in his 
e-A23 form. The airport will contact Nate 


once they find his missing baggage. 


Henry gives Nate a paper A23 receipt to 
show the officer at Egress. 


Step 8: Egress 


Nate Delacour enters Egress Lane #4. 


Step 1: View Traveller State on Screen 

Frank Potter sees that “Nate Delacour” is a 
who should have “4” piece 

of luggage. However, Frank only sees 2 pieces 

of luggage on Nate’s trolley. 


Step 2: Direct Nate out of the Lane 


Frank gently waves Nate out of Lane #4 and 
asks him to wait. 


Step 3: Call Roving BSO 


Frank calls over BSO Esther Diggory and 
explains the problem. 


Esther Diggory 
(Egress Roving BSO) 
Step 1: Talk to Traveller 


BSO Esther Diggory asks Nate Delacour why 
he only has 2 bags. 


r 


Step 9: Show A23 receipt 
Nate Delacour shows his A23 receipt to BSO 
Esther. 
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BSO Esther reviews Nate’s paperwork. 


Step 3: Direct traveller 


BSO Esther decides to let Nate proceed. 
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Nate enters Canada. 


Scenario #3 


Marie Bennet 
(International traveller) 
Step 1: Profile Creation 


Marie opens the MyTravel app and creates 
her profile: 


User Email address 

PIN: 6666 

Scan the picture page of your 
passport (make sure the bottom 2 
lines of the page are in the scan 
window of the camera) 

Take 1 Profile photo (your 
normal, relaxed face) 

Take 7 RBR photos {various facial 
gestures) 

Apply for eTA 

Pay with your fake credit card: 
4111111111111111 


Marie is ready to fly! 


Step 2: Check In 


Marie Bennet “Checks In” at the airport: 


Scan the picture page of your 
passport. 

Flight: AC204 

Bags: 2 


Jake Austen 


(Mobile Enforcement Command) 


Step 1: Assign Roving BSO to a Zone 


Jake assigns BSO Andria to the Primary zone 
of the airport. 


BSO Andria Bingley 
(Roving BSO) 
Step 1: Sign In to MEC system 


BSO Andria Bingley signs into the Mobile 
Enforcement Command system. 


7 


Step 2: Tap “Roving” button on watch 


Step 3: Walk to Primary 


Step 4: Check In to Primary 


On arrival, BSO Andria “checks in” to the 
Primary zone using the mobile device. 


Page 24 of 35 
PROTECTION » SERVICE + INTEGRITY 


A-2018-09320-001171 


à Immigration, Refugees Immigration, Réfugiés 
and Citizenship Canada et Citoyenneté Canada 
s.1 6(1 Xc) Information disclosed under the Access to Information Act 


L'information divulguée en vertu de la loi sur l'accès à l'information 


Canada Border Agence des services 
Services Agency frontaliers du Canada 


Border Services EI 


* PRESS THE “FLIGHT DEPARTURE” BUTTON 
FOR FLIGHT AC204. 


FAR Services frontaliers 


Step 3: Prepare e-Dec 
In flight, Marie prepares her e-Declaration: 


Exemptions: $0 
Flight: AC204 
Take RBR Photo 


* PRESS THE “FLIGHT ARRIVAL” BUTTON 
FOR FLIGHT AC204. 


Step 4: Submit e-Dec 


Marie walks off the plane and can now press 
the “Submit” button on her e-Declaration. 


The MyTravel app notifies Marie that she has 
to go to a Primary kiosk for photo validation. 


However, Marie sees that her mobile phone 
is running out of power, so she finds an 
outlet in the wall somewhere in the Arrival 
zone and sits down next to it for an hour.... 


Step 1: See Alert 


The Orchestrator notifies Mobile 
Enforcement Command that “Marie Bennet” 
is a traveller who should be somewhere in 
the Primary zone. : 


Step 2: Forward the Alert to Roving BSO 
Jake Austen forwards the alert to BSO Andria 
Bingley. 


Step 5: See the Alert 


BSO Andria Bingley receives the alert on her 
watch and then reviews the traveller 
information on her mobile device. 
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Step 6: BSO Andria walks around the Primary 


zone looking for Marie Bennet. 


BSO Andria eventually finds Marie Bennet. 


Step 7: BSO Andria presses the “Engaged” 
button on her watch. 


Step 7: Scan traveller’s passport 
BSO Andria talks to Marie Bennet. 
BSO Andria asks for Marie’s passport and 
scans it with her mobile device. 
Marie Bennet hands her passport to BSO 


Andria. 


BSO Andria now reviews Marie’s travel 
profile and information on her mobile device. 


Step 8: Override time alert 

BSO Andria overrides the time problem using 
her mobile device. BSO Andria tells Marie to 
present herself at a Primary kiosk as soon as 


possible. 


Marie Bennet unplugs her mobile phone and Step 9: BSO Andria presses the “Roving” 


walks towards a Primary kiosk. button on her watch. 


Step 5: Primary Kiosk 


Marie Bennet goes to kiosk #11 in the 
Primary zone. At the kiosk: 


Scan the picture page of your 
passport 

Scan passport chip. 44 Close 
your passport and hold it next to 
the back of the kiosk (keep it still 
for 10 seconds as the reader tries 
to find the chip) 

Take RBR photo 


The MyTravel app notifies Marie that she can 
proceed to the Baggage Carousel zone to pick 
up her 2 bags. 
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Step 6: Baggage Carousel 


FAR Services frontaliers 


Marie Bennet picks up her 2 bags. 
Step 7: Egress 
Marie enters Egress Lane #4. 
Jonathan Darcy 
(Egress Security Officer) 
Step 1: View Traveller State on Screen 
Jonathan Darcy sees on his screen that 


“Marie Bennet” is a who 
should have “2” piece of luggage. 


Jonathan Darcy decides to let Marie proceed. 


Marie enters Canada. 


Scenario #4 


Ruth Organa Luke Walker 
(Canadian traveller) (Egress Security Officer) 
Step 1: Profile Creation 


Ruth Organa opens the MyTravel app and 
creates her profile: 


User Email address 

PIN: 6666 

Scan the picture page of your 
passport (make sure the bottom 2 
lines of the page are in the scan 
window of the camera) 

Take 1 Profile photo {your 
normal, relaxed face) 

Take 7 RBR photos {various facial 
gestures) 
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Step 2: FAQ 


28 Services frontaliers 


Ruth has a question about duties and fees, so 
she opens the FAQ section of the MyTravel 
app and easily finds the answer. 


Step 3: Request travel history 


Ruth needs a copy of her travel history, so 
she goes to the Passage History section of the 
MyTravel app and requests it to be sent to 
her. 


Step 4: Check In 
Ruth Organa “Checks In” at the airport: 


Scan the picture page of your 
passport. 

Flight: AC336 

Bags: 1 


* PRESS THE “FLIGHT DEPARTURE” BUTTON 
FOR FLIGHT AC336. 


Step 5: Prepare e-Dec 


In flight, Ruth prepares her e-Declaration: 


Exemptions: $0 
Flight: AC333 (an error) 
Take RBR Photo 


Step 6: Fix e-Dec 


While in flight, Ruth remembers that she 
accidentally put the wrong Flight # in her e- 
Dec. So she opens the app, presses the 
“Home” button, and then re-does her e- 
Declaration (with the right Flight # this time!) 


Flight : AC336 


* PRESS THE “FLIGHT ARRIVAL” BUTTON 
FOR FLIGHT AC336. 


Step 7: Submit e-Dec 


Ruth walks off the plane and can now press 
the “Submit” button for her e-Declaration. 
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The MyTravel app notifies Ruth that she has 
to go to Primary for photo validation. 


28 Services frontaliers 


Step 8: Primary Kiosk 


Ruth Organa goes to kiosk #5 in the Primary 
zone. At the kiosk: 


Scan the picture page of your 
passport 

Scan passport chip. LB Close 
your passport and hold it next to 
the back of the kiosk (keep it still 
for 10 seconds as the reader tries 
to find the chip) 

Take RBR photo 


The MyTravel app notifies Ruth that she can 
proceed to the Baggage Carousel zone to pick 
up her bag. 

Step 9: Baggage Carousel 

Ruth Organa picks up her bag. 


Step 10: Egress 


Ruth enters Egress Lane #7. 


Step 1: View Traveller State on Screen 
Luke Walker sees on his screen that “Ruth 
Organa” is a‘ who should 


have “1” piece of luggage. 


Luke Walker decides to let Marie proceed. 


The MyTravel app reminds Ruth that her 
passport is about to expire. 


Ruth opens the My Travel app and spends a 
few moments completing her passport 


renewal application. 
She: 


Answers some standard questions 
Takes a new passport photo 

Pay with your fake credit card: 
4111111111111111 
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She then presses the submit button. 


The MyTravel app gives Ruth a QR code that 


she can present at a local passport office if 
there are any questions about her 
application. 


Scenario #5 


BSO Devin Kent Thérése Wayne Jack Napier 
& (Mobile Enforcement Command) & 
BSO Sylvie Prince Veronica Pennyworth 
(Roving BSOs) & 
Peter E. Nigma 
(Travellers) 
Step 1: Both BSOs sign into the Mobile Step 1: Profile Creation 
Enforcement Command system. 
Step 1: Assign Roving BSOs to their Zones The travellers open the MyTravel app and 
create their profiles: 
> Therese assigns BSO Devin to the Baggage 
Carousel zone. . User Email address 
PIN: 6666 
— Therese assigns BSO Sylvie to the . Scan the picture page of your 
passport (make sure the bottom 2 
lines of the page are in the scan 
window of the camera) 
Take 1 Profile photo (your normal, 
relaxed face) 
Take 7 RBR photos {various facial 
gestures) 


Intermediate zone. 


Step 2: The BSOs tap “Roving” button on 
their watch 


International travellers: If you have a non- 


Canadian passport, you will also have to do 


Step 3: Walk to their zone the eTA part of enrolment. 


Apply for eTA 
Pay with your fake credit card: 
4111111111111111 


The travellers are ready to fly! 
Step 4: Check In to Zone 


On arrival, the BSOs “check in” to their zone 
using the mobile device. 
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..BSOs Roving! Step 2: Check In 


FAR Services frontaliers 


The travellers “Check In” at the airport: 


Scan the picture page of your 
passport. 
Bags: 1 
Different Flight numbers: 
- Jack Napier: AC882 
- Veronica Pennyworth: AC919 
- Peter E. Nigma: AC314 
* PRESS THE “FLIGHT DEPARTURE” BUTTON 
FOR FLIGHTS AC882, AC919, and AC314. 
…Roving! 


Step 3: Prepare e-Dec 


In flight, the travellers prepare their e- 
Declarations: 


Exemptions: $0 
..Roving! . Take RBR Photo 


Different Flight numbers: 
- Jack Napier: AC882 
- Veronica Pennyworth: AC919 


- Peter E. Nigma: AC314 


* PRESS THE “FLIGHT ARRIVAL” BUTTON 
FOR FLIGHTS AC882, AC919, and AC314. 
..Roving! 


Step 4: Submit e-Dec 


The travellers walk off the plane and can then 
press the “Submit” buttons on their e- 
Roving! Declarations. 


The MyTravel app notifies the travellers that 
they will have to go to a Primary kiosk for 
photo validation. 
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Step 5: Primary Kiosk 


TRAVELLER INSTRUCTIONS: 


Jack Napier: Go to kiosk #13 in the Primary 
zone. At the kiosk: 
„Roving! 

Scan the picture page of your 
passport 
Scan passport chip. L Close 
your passport and hold it next to 
the back of the kiosk (keep it still 
for 10 seconds as the reader tries 
to find the chip) 
Take RBR photo 


Veronica Pennyworth: Wait for a while in the 
Primary zone. You will have a “broken RBR” — 
instructions will follow later in this script. 


Roving! 


Peter E. Nigma: Just walk around the Arrival 
Hall. Do not go to a kiosk. 


The MyTravel app notifies Jack Napier that he 
can proceed to the Baggage Carousel zone to 
pick up his suitcase. 


BSO Devin (Baggage Carousel zone) sees Jack 
Napier and is suspicious of him... 


Step 5: BSO Brendan presses the “Engaged” 
button on his watch and approaches Jack 


Napier. 
Step 6: Scan traveller’s passport 


BSO Devin asks for Jack Napier’s passport and 
scans it with his mobile device. 


BSO Devin now reviews Jack’s travel profile 
and information on his mobile device. BSO 
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Devin suspects that Jack is bringing drugs into 
Canada, so he says goodbye to Jack and then 


FAR Services frontaliers 


uses his mobile device 

BSO Devin can pick 
from a dropdown list of reasons for the 
referral. 


Jack Napier should now be diverted at Egress 
and sent to Secondary. 


Step 7: BSO Devin presses the “Roving” 
button on his watch. 


Step 2: Re-Assign BSO Devin to the Primary 
Zones 


— Therese assigns BSO Devin to the Primary 
zone. 


Step 8: BSO Devin acknowledges the re- 
assignment, and starts walking to Primary. 


Step 9: BSO Sylvie Prince sends out a Red 
Alert (call for help). 


Step 10: BSO Devin taps “Responding” on his 
watch and starts running towards BSO Sylvie. 


He finds her and helps her subdue a drunk 
traveller. 


Step 11: BSO Devin taps “Roving” and 
continues walking towards the Primary zone. 


Step 12: Check In to Primary Zone Veronica Pennyworth: Go to kiosk #6 in the 
Primary zone. At the kiosk: 

On arrival, BSO Devin “checks in” to the 

Primary zone using the mobile device. . Scan the picture page of your 
passport 
Scan passport chip. L Close 
your passport and hold it next to 
the back of the kiosk. 
Take a “bad” RBR photo (ie. look 
down or have someone else take 
the photo — we need you to break 
the photo-matching system) 
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The bad/broken RBR issue should alert the 
Orchestrator of a problem. The Orchestrator 
will then forward the alert to the Mobile 
Enforcement Command. 


FAR Services frontaliers 


Step 3: See Alert 


The Orchestrator notifies Mobile 
Enforcement Command that there is a 
problem verifying a traveller at kiosk # 6 in 
the Primary zone. 


Step 4: Forward the Alert to Roving BSO 


Therese Wayne forwards the alert to BSO 
Devin in the Primary zone. 


Step 13: See the Alert 


BSO Devin Kent receives the alert and walks 
to kiosk #6. 


Step 14: BSO Devin presses the “Engaged” 
button on his watch. 


Step 15: Scan passport 


BSO Devin scans Veronica Pennyworth’s 
passport and reviews her travel profile and 
information on his mobile device. 


Step 16: Override RBR issue Veronica Pennyworth can now continue with 


her Primary kiosk tasks. 
BSO Devin overrides the RBR problem using 
his mobile device. 


Step 17: BSO Devin presses the “Roving” on Peter E Nigma is walking around the Primary 
his watch. zone but has not gone to a kiosk yet. 


Step 5: See new alert 

Therese Wayne sees an alert from CBSA 
Targeting. There is a national security lookout 
for “Peter E. Nigma” — a traveller who has 


just flown in to the airport. 


Step 6: Forward the Alert to ALL Roving 
BSOs 


Therese Wayne forwards the alert to all 
Roving BSOs in every zone. 
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Step 18: Review Alert info 


FAR Services frontaliers 


BSO Devin and BSO Sylvie review all the 
known information about the traveller. 


Then they keep roving while looking for the 
traveller. 


Step 19: Finding the traveller 


BSO Devin finds Peter E. Nigma in the 
Primary zone, so he takes ownership of the 
alert (which cancels it for BSO Sylvie). 


Step 20: BSO Devin presses the “Engaged” 
button on his watch and goes to talk to Peter 
E. Nigma. 
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Chain of Trust (CoT) 
Exercise 4 - Outcomes and Notes 


Services frontaliers 


maaa 7 D NM 


Exercise 4 Summary Report: 

The Chain of Trust project completed Exercise 4 at the CBSA Science and Engineering facility from March 
6" to 8, 2018 (with two pre-exercise testing sessions held in the weeks beforehand). This was the 
fourth and final exercise planned for the Chain of Trust prototype project. 


Exercise 4 featured several notable advancements from Exercise 3, which had been held in December 
2017. Of note: 
e the project team ran larger volumes of travellers through the system concurrently and assessed 
the outcomes; 
e the project team tested indoor positional technology for the CBSA Mobile Enforcement app 
which could aid Roving Border Service Officers (BSOs) to localize each other within the airport; 
e the IRCC and CBSA predictive analytics teams tested three new scenarios for collaboratively 
identifying high-risk travellers; 
e IDENTOS collected usability feedback from the volunteer actors (both travellers and Roving 
BSOs). 


As with previous exercises, the planning for the exercise consisted of bi-weekly teleconferences of 
micro-team leaders, to establish planning, brief on progress and collaborate on remaining challenges. 


Exercise 4 

The core systems for Chain of Trust had been developed and built in large part by the end of Exercise 3, 
and they continued to function as envisioned during Exercise 4. This allowed the project team to focus 
on other aspects, such as increasing the volume of the testers, assessing additional scenario cases not 
previously covered, and gathering usability feedback from the testers. 


To assess the system’s handling of larger volumes of travellers, approximately 15 CBSA volunteers were 
engaged to play the role of travellers moving through the system at the same time. This testing was 
conducted during the full day of March 6" and the morning of March 7“. Despite a few problems, such 
as issues with bandwidth and messaging, the system generally performed as expected. 
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On the afternoon of March 7”, 2018, a demonstration of the system was provided for IRCC senior 
management, including several directors, as well as the Director General of IRCC’s Operations, 
Performance and Planning Branch, and its Assistant Deputy Minister Operations. 


The final day of the exercise, March 8", consisted of a project wrap-up discussion — assessing progress 
and remaining challenges, and discussing next steps and the project close out. A summary of this 
discussion is provided below: 
e Progress: 
o The project team noted that the modular sub-systems within Chain of Trust (largely 
“plug and play”) really allows for flexibility in terms of further development. The team is 
very happy with how well this has worked out. 
The Mobile Enforcement app and smart watch were very well received by testers. 
The use of predictive analytics was more clearly identified in this exercise thanks to new 
scenarios and additional displays showing the back-end risk scores. 


e Challenges: 


o It was noted during discussion that additional diagnostic interfaces would have been 
useful for troubleshooting the Orchestrator. 
o It was noted that the user experience feedback may be somewhat biased due to the 
step-by-step instructions that were provided to the travellers early each morning. 
However, these instructions were ultimately the fastest way to get the testers on the 
floor. 
e Next Steps: 
o Mike Haymes and Naomi Barrett will write the Exercise 4 report and distribute to the 
project team for review. 
o Partner contracts are currently being amended to end on March 31°, 2019 in order to 
include one year of prototype demonstrations. 
e Project Close: 
o [RCC will write the project close out documentation for CSSP. 
o Project partners must all submit their final reports to Naomi Barrett by March 25", 
2018. 


Page 2 of 9 
PROTECTION » SERVICE + INTEGRITY 


A-2018-09320-001184 


à Immigration, Refugees Immigration, Réfugiés 
and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 


L'information divulguée en vertu de la loi sur l'accès à l'information 


Canada Border Agence des services 
Services Agency frontaliers du Canada 


Border Services $ 


Usability Testing 
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Usability assessment played a prominent role in Exercise 4. Chain of Trust engaged IDENTOS to provide 
an assessment of the usability feedback collected from volunteer testers and BSOs during Exercise 4. 
With the aim of capturing the widest range of feedback possible, the usability testing took place in two 
parts: 


e Observation and interacting with testers as they moved through the system: Several project 
team members were assigned to move among the travellers and Roving BSOs during Exercise 4 
with the aim of identifying and capturing the issues that troubled or confused the testers. These 
note-takers wrote down any thoughts from the testers (miscommunications, room for 
improvement, tips, etc) as they occurred. 

e Tester surveys: At the end of each testing day, each tester was asked to complete a user survey 
designed to help the project team understand overall satisfaction, and point out places where 
the testers identified room for improvement. 


While this strategy worked reasonably well, there were some concerns about providing step-by-step 
instructions to the testers as a potential source of bias for the feedback received. 


Initial recommendations/observations from the usability assessment include: 


e Both travellers and BSOs expressed appreciation for how the apps could make their lives easier 
in the future. 

e Ensure travellers and BSOs have a clear view of where they are in any process. Travellers 
sometimes lost track of what they needed to do and when (e.g., to submit their e-Declaration 
upon arrival). BSOs were unsure whether their actions actually took effect. 

e Clarify messaging. Ensure messaging is written to be readable by the uninitiated; replacing 
jargon and acronyms with more meaningful copy, even if it’s longer, or ensuring that the 
phrases used match their real-world context. Adjusting the messaging can help users make 
decisions and take actions with confidence. 

e Importance of ensuring the needs of users are met, as opposed to simply ensuring all 
components can successfully communicate with each other. Especially in the case of BSOs, 
whose needs for use weren’t fully met by the existing app. 


As anext step, IDENTOS will synthesize feedback from the user surveys and note-takers and compile 
usability recommendations for the project. These recommendations will be made available in the final 
report from IDENTOS. 
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Micro-Team reports: Observations and notes from Exercise 4 
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To support this report, each micro-team leader was asked to provide notes from Exercise 4 on their 
subsystem - explaining which pieces worked, where they experienced challenges, and describing 
opportunities for further development of their subsystems. Summaries of these notes are provided 
below: 


> TRAVELLER MOBILE APP (“MY TRAVEL” APP): 
A majority of the effort since the previous exercise focused on refinement and completion of 
communications with other system components. The creation of distinct development environments in 
the AWS Cloud in January had allowed all partners to work on their system components while 
maintaining a clean demo environment that was available to show at any time. 


As a result of the efforts since Exercise 3, the fourth exercise saw data flowing between each 
component system without the need for stubs or hard coding, which was one of the key objectives of 
the project. 


A newer model of IRCC Predictive Analytics was integrated with the MyTravel service. This allowed 
scenarios to be tested with travellers fitting certain profiles to trigger various scores that fed into the 
Orchestrator and could flag them for Mobile Enforcement upon arrival. 


The final exercise highlighted some areas where more fine-tuning of the MyTravel app would be 
needed, such as: 
- Notifications to the traveller: Need to ensure it is clear whether the traveller is to take an action 
as a result of the message or if it is for information purposes only. 
- More information about the outcomes and instructions — make it more intuitive. 
- The travellers didn’t seem to like taking selfies. There is a need to find a way to get photos of the 
traveller (as required for the Renewable Biometric Reference) that does not involve the traveller 
taking selfies (e.g. video). 


There are also some places in the flow where dependencies ought to be strengthened. For example, the 
MyTravel app allows travellers to submit their e-Declaration upon arrival, but there is currently nothing 


that keeps them from going through egress without submitting it. This resulted in some of the system 
messaging —Orchestrator, Flow Control and MyTravel — to be out of sequence. 
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Sequencing and receipt of messages between the component systems was an area that would require 


further analysis and testing as there were issues with lags when there were larger number of travellers. 
Network configuration has already been identified as a first area to address. 


> KIOSKS (CHECK-IN, PRIMARY & INTERMEDIATE) 
The functionality kiosk didn’t change significantly from Exercise 3 although there were a few 
modifications: 


- the Check-In kiosk added an additional line for number of travellers. 


A gap was identified when travellers at the Primary kiosk fail to complete their task properly (whether 
user error or system error). The kiosk tells the traveller that a BSO will come to assist them momentarily, 
however there is no mechanism currently in place that communicates this to Mobile Enforcement 
Command. 


> CBSA MOBILE ENFORCEMENT APP AND DISPATCH SYSTEM 
Overall, the CBSA Mobile Enforcement app and dispatch systems worked as expected. Several new 
advancements were incorporated and tested during Exercise 4. 


Bluetooth beacons were installed in the “airport” to test a new feature of the Mobile Enforcement app: 
officer localization within the arrivals hall. Testers reacted positively to the idea. However, there were 
some challenges relating to the API for the Bluetooth beacons, which weren’t fully exposed for Android 
(the iOS API seemed to have more capabilities). 


A traveller image was also added to the smart watch which would allow Roving BSOs to visually identify 
a traveller who requires intervention. However, network issues sometimes precented the sending and 
receiving of messages from the Orchestrator. 


One area that had been discussed by the project team, but was ultimately not incorporated into the 
project was the deployment of Google Glass for the Roving BSOs. This was not included due to limited 
support and documentation for the technology. Further research into heads-up displays will continue. 


Overall, discussions around the Mobile Enforcement app and smart watch were very positive. In fact, 
there was some talk about using the smart watch even more in order to allow the BSOs to be hands- 
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free. In regards to the app on the smart phone, there were some discussions about traveller data privacy 


and ways to prevent others from seeing the information that a BSO is viewing on the screen. 


There was also some discussion around identifying what information is the most pertinent to display on 
the Mobile Enforcement Command screen, and what information could be hidden. 


> RENEWABLE BIOMETRIC REFERENCE (RBR) 
All components for the RBR were complete and working for Exercise 4. The RBR server was up and 
running, which can automatically generate a client’s RBR, verify an RBR, and generate and verify the e- 
declaration “biometric signature.” The service had no problem with the increased load of travellers 
during the exercise. 


A slightly higher strictness of matching algorithm was used than for Exercise 3. During the exercise, 
some travellers failed their RBR validations multiple times due to not having enough variability in their 
selfies. It was shown that it is important that the users are aware that they need to have variability in 
their selfies, for example, by singing or talking while taking the selfies. The more uniform lighting during 
the exercise helped for the accuracy of the RBR. 


> ORCHESTRATOR 
In general, the Orchestrator performed well in Exercise 4. New user displays were added that identified 
traveller risk scores as they passed through the system. This proved very valuable in demonstrating the 
back-end risk assessment taking place throughout the travel continuum. 


New rules were added to handle travellers with high-risk, as assessed by predictive analytics. These rules 
would flag travellers who’s risk scores exceeded a particular threshold and send these flags to the 
Mobile Enforcement Command for review. 


As with Exercise 3, the Orchestrator had some issues with message lag and the message order being 
mixed up, possibly tied to bandwidth limitations in the testing environment. These issues caused 


significant problems for the sub-systems depending on the Orchestrator and demonstrated the critical 
importance of message timing and Orchestrator dependability. 
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Some additional areas for possible improvement were noted: 


Services frontaliers 


- It was noted that additional diagnostic interfaces would have been useful for troubleshooting 
the Orchestrator. 

- CBSA staff noted that they would like the ability to change/add policies to the logic depending 
on location (eg. airport superintendant could experiment with changing the thresholds for 
Secondary referrals). The ability to simulate the effect of the policy changes and examining the 
data (based on the past traveller data at their airport) would be hugely beneficial. 


> IRCC PREDICTIVE ANALYTICS 
In past scenarios, a simple dummy script had been used to demonstrate the flow of information, with a 
separate unconnected laptop available to demonstrate the machine-learning scoring model built for the 
project. For Exercise 4, IRCC Predictive Analytics deployed a predictive analytics scoring model to the 
back-end of the WorldReach mobile application, enabling scoring of eTA travellers within the general 
dataflow. 


The scoring and communication functioned as expected. High-risk travellers were given high scores by 
the IRCC model and were subsequently flagged by the Orchestrator for BSO intervention. 


There were some issues in the deployment of the scoring script to the WorldReach server. 


In the project wrap-up meeting, there was some discussion about the best way to provide risk scores to 
human officers without creating information overload or potentially biasing their judgement. If the 
model could also provide additional information about why a person scores high, this would be useful to 
human officers. 


> CBSA PREDICTIVE ANALYTICS 


To demonstrate the use of predictive analytics within the chain of trust system, the CBSA PA engine was 
configured and trained to score scenarios to trigger 1 Orchestrator rules. 
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The CBSA predictive analytics believes that additional opportunities could be included in Chain of Trust 
in the future, such as: 


- including a graphical user interface that can interact with BSOs to translate human heuristics to 
machine configurable heuristic; 

- refining the system to periodically update the model, taking historical data transactions as an 
input and delivering updated and refined model parameters as the output. 


> FACE RECOGNITION/TRAVELER FLOW 
Flow Control Lanes worked at a proof of concept level both on its own and when integrated with Chain 
of Trust system. 


There were some issues relating to delayed messages from the Orchestrator, in some cases minutes 
after they were sent. Some faces were not detected in the lane. 


Several advancements were completed for Exercise 4, including: 
e New interfaces to Orchestrator (Minio) 
e Re-architected to have Centralized FFS database for all lanes and separate lane controllers 
e Optimized state machine 
e Auto-reset for each lane so if there is an issue with use of the lane, lane will reset to normal 
state (this is to facilitate demonstration when complete system is not in place) 
e Adjustments to UX based on feedback from Exercise 3. 


Although not in the initial scope, additional flow control lane performance testing was planned following 


the completion of Exercise 4. This will use volunteers to assess the quality of face recognition 
performance and lane transit times. CBSA was not able to integrate a new match server for BSOs. 


The issue of what to do in the event of a missed face-capture was 
also discussed as part of the exercise wrap-up. 
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> PASSPORT RENEWAL APPLICATION s.16(2) 
A mobile passport renewal app, built by WorldReach under Chain of Trust 


streamline the passport renewal process, was also demonstrated as part of 
Exercise 4. The app was built on the technology developed as part of the CSSP-funded Faster PrivBio 
project. Discussion is ongoing about whether the photos taken by mobile phones are capable of meeting 
ICAO specs for the purposes of inserting on a passport. A copy of the app was provided to IRCC for in- 
house demo purposes and the app was included as part of the Chain of Trust demo given to IRCC senior 
management during Exercise 4. 


Further Testing and Demonstration 

Although system development has completed, CBSA has arranged to maintain the dedicated project 
space at 79 Bentley Avenue in Ottawa for a one-year period post-project to allow for further 
demonstration and internal testing by CBSA. Project partners have agreed to provide a technology 
training session on their individual portions of the project to enable CBSA staff to effectively use the 
technology. The testing is expected to be valuable for CBSA in their strategic visioning exercises over the 
coming year. 


CBSA is working out details relating to how any future demonstrations would be delivered following the 
completion of the Fiscal Year. 


Page 9 of 9 
PROTECTION » SERVICE + INTEGRITY 


A-2018-09320-001191 


à Immigration, Refugees Immigration, Réfugiés 
and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 
s.16(1)(b) o 


s.16(2) 


CHAIN OF TRUST : COLLABORATIVE RISK ASSESSMENT FOR AIR TRAVELLER PROGRAMS - Concept of Operations 
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Prepared by OPMB-PMU- 
DPAT S PY 


Speaker: Stephanie Kirkland 


Predictive analytics have proven value in private industry and government 


Predictive Analytics are a key piece of the modernization strategy 
DPAT is working closely with partners: SPO, PID, CPR, IR and OMC 


CIC is at the very beginning of using predictive analytics 
We are catching up on other countries (USA and Australia for example) and other 
departments 


Expectations need to be set carefully 
Predictive analytics can be transformative for an organization, 
but it takes a long time to put everything in place 
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Predictive Analytics ~ Overview 


What is predictive ansiytics 

Pronos aivalylice at CIC 

Wry is Predictive analytics uaportant for CL 

Why Predictive Analytics in Querstions 
Engagement of Key Stakeholders 

Predictive Analytics and Policy sector 

Projects update — 

Example-Predictive Analytics Draft Model for VR-extension 
DPAT. Next steps 

Anne Predictive Analytics Anprnach and Method 
Anne: Data and Predictive Analytics Roadmap 


Anneri Key steps lor Success 
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What is Predictive Analytics? 


» Predictive analytics is the process of discovering patterns in historical 
data and behaviours to infer the likelihood of future events and to 
predict probable outcomes 


Drawing on a combination of... Benefits 


Computer Sciences » Formally substantiates business decisions made hy intuition ar gut 


feeling 
Statistics $ . 
Combine operational businessknowledge and expertize with 


Gperational expertise statistical analysis 
Specialized Software 
Heips makimicing the use of resources 


Drives smarter decision making By axtracting insights fram tha vast 
quantty of dats available in government agencies. 


Canada 


Speaker: Stephanie Kirkland 


Explains definition and how it works. Also talk about concrete benefits in any business 
doing predictive analytics (Walmart, for example, or Google, that both use that now in their 
business to save money, profile their consumers and improve resource allocations) 


The benefits of combining operations expertise with empirical data and statistical analysis 
to build a predictive model: 


& 


* increasing processing capacity 


& 


* Facilitate program integrity exercises 
* Deliver optimal client service 


Forecasting vs Predictive analytics: 


How will predictive analytics help to detect fraud?: 
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Predictive Analytics at CIC have two important goals: 


ie oe, men, 


Speaker: Stephanie Kirkland 


1. Enable CIC’s modernization agenda: 


e By reducing the labor intensity of application triaging and 
processing as well as minimize the time to a decision for a 
significant number of cases. 


2. Help to detect and reduce fraud and contribute to program integrity 


Predictive Analytics at CIC have two important goals: 
1. 


2. 


A-2018-09320-001196 


Predictive analytics at CIC will: 
i, Enable CiC’s modernization agenda 


2. Help ta detect and reduce fraud and cantribute to program integrity 


3 Certedute ts longterm: program eusluetion and to battar faciifiate policy development, 


4. Reip CiCto cope with increased volurne of applications 
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Lead: OPMB- PMU 


* Established axpe 
volumes of actual and histarical data, 


* Network wide experience in workload distribution in collaboration with the network and sex tar. 


* 
: 
* OPM js the lead for Data Governance in the departinent, in partnership with RRE. 
2. Key partners and clients: 
+ OMC: Program expertise {TRPD, PRED] and progrern integrity (PID) 
* SAO: Project brads ard clients 
* CER (including regians when needed) and IR: Experts in processing and subject mattor experts far intelligence 
gathering 
SIME: support IT systems and data 
Other departments and jurisdictions; for lessons learned, best practises and information sharing, 
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Our stakeholder engagement” fs three told 
1. Bi-weekly working group meetings with partners: 
» OPMB/ DPAT identifies and meets regularly with partners and stakeholdars ta collect business 
requiraments for building predictive modals that ara aligned with CIC business and with 
madernizationagends. 


Working groups are by line of business and by projects (IN orciact ~ JEC, Extension, Students, eTA, 
and Express Entry} 


2. Stakeholder engagement sessions 
» The purpose of early engagementis to build relationships, explain what Predictive Analytics is and its 


benefits; and te establish protocols far delivery and communication between the teams. 


3. Engagement and sharing best practices with other departments and other countries; 


o We Rave had meetings with many groups of CBSA; we chair a quarterly Date Analytics Coalition 
regrouping 14 government departments and we have engaged with foreign governments such as the 
to share techniques and best practices. 


“our stakeholder engagement started April 2014. 


Speaker: Marie-Josee Dorion 


> Our stakeholder engagement* is three folded: 
1.Bi-weekly working group meetings with partners 
_ Every DPAT project (models) starts with a clear definition of business goals 
and a partnership is formed with the line of business 
-DPAT Discuss findings with the business on regular basis for validation and 
to combine qualitative business knowledge with empirical statistical research 
2. Stakeholder engagement sessions 
-bi-monthly stakeholder engagement sessions to discuss findings with the 
business on regular basis for validation and to combine qualitative business 
knowledge with empirical statistical research 
-Program experts, policy and administrative teams need to be involved to 
ensure successful project delivery 
3. Engagement and sharing best practices with other departments and 
other countries; 
-Held numerous meeting with CBSA 
-Initiated a collaborative effort with other government departments and 
chairs a Data Analytics Coalition with key government departments and 
agencies 


A-2018-09320-001199 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


s.15(1)(i)i) 


A-2018-09320-001200 


* Predictive Analytics can support a specific decision task or policy by analysing historical data 
and identify meaningful patterns 


Contributions from the Policy Sector to Predictive Analytics: 


Provides input, guidance and support to senior management on the develapment of a 
robust predictive analytics models 

Provides support to senior management participation in various consultations 
committees and working groups 

Provides regulatory and analytical policy advice and support 

Challenges the predictive analytics approach in the pursuit of rigour and integrity 


Canada 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


s.16(1)(b) 


s.16(2) 


A-2018-09320-001201 © 


à Immigration, Refugees Immigration, Réfugiés 
and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 


LiSoAGE }bihuée en vertu de la loi sur l'accès à l'information 


s.16(2) 


Projects Update 


1- Express Entry 

OPAT analyred agplications processing to identify the mast important indicators and build a mariel with the goal ot 
allowing sutometed distributional applications, 
Express Entry-phase L Preliminary model done and business rules extracted. 
Express Entry-phase 2: will start in a few months, when more Express Entry applications have been processed 
and finalized, 

o The goal is to have a solid model that we can deploy into production tor auto-triaging of applications 
{tentative date tor now is Spring AGE) 


_ 


Speaker: Marie-Josee 


1- Express Entry: 


1- Express Entry-phase 1: COMPLETED 
*DPAT developed models for FSW, CEC and PN and delivered 


Business rules for launch of Express Entry 


*DPAT is working with CPR and Ops to use the business rules a 
level deeper by using GCMS Answers to create dashboards that 
will allow applications to be grouped and assigned, using the 
dashboard. 


2- Express Entry-phase 2: will start in a few months 


*DPAT is now waiting for few months to collect real data and 
validate or re-built the models developed based on proxy data. 
Once the data is available, Express Entry phase-2 will start in June 
2015. 
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Express Entry auto-triaging model will be deployed in spring 2016. 


2. eTA phase-2: will be starting in Oct 2015 (or even later now, 
with the new date) 


1.TR Pilot: for deployment in the Spring/ June 2016. 
1.1. ICE 
IEC Working Groups held with stakeholders and draft model and GCMS 
Answers case-statements was delivered to OSC. Currently participating in 
INP IEC Phase 1 implementation (the digitization of IEC). 
DPAT will rebuild the model just prior to launching the Phase 2 pilot in 
2016. 


A-2018-09320-001203 


Immigration, Refugees Immigration, Réfugiés 
| and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Informal 
L'information divulguée en vertu de la loi sur l'accès à l'information 


s.16(1)(b) 
s.16(2) 


1.2. VR-Extension 


VR-extension project started in Dec 2014 and is finalized in February 
2015. 

DPAT delivered Predictive Analytics Model to SPO and OPS. 

1.3. STUDENT 

_DPAT is starting the working groups with stakeholders to develop 
predictive model for auto-triaging 


2.Full TR 


TRV_and TFW (including LCP) 
-DPAT will start the working groups with stakeholders to develop 


predictive models in Spring 2015 
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Projects Update (con't) 


1- TR Pilot: for deployment in June 2016. 
+ IEC: Draft model COMPLETED 
+ VR-EXTENSION : Draft model COMPLETED 
» STUDENT : started in February 2015 
3 Draft model under developeri 
The models developed during 2014 and 2015 for the TR Pilot wili be retested and validated with new applications 
in the months before launch, 


2- Full TR: for deployment in January 2017 
+ The “full TRE includes : 
oo Visitors (IRV), Femupsrary Foreign Workers {IHW}, including Live-in Caregiver (LUS 


+ TRY and TEW (including LCP): Development of predictive models will start in Spring 2015 


Speaker: Marie-Josee 


1.TR Pilot: for deployment in the Spring/ June 2016. 
1.1. ICE 
IEC Working Groups held with stakeholders and draft model and GCMS 
Answers case-statements was delivered to OSC. Currently participating 
in INP IEC Phase 1 implementation (the digitization of IEC). 
DPAT will rebuild the model just prior to launching the Phase 2 pilot in 
2016. 
1.2. VR-Extension 


VR-extension project started in Dec 2014 and is finalized in February 
2015. 

DPAT delivered Predictive Analytics Model to SPO and OPS. 

1.3. STUDENT 

-DPAT is starting the working groups with stakeholders to develop 
predictive model for auto-triaging 
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TRV_and TFW (including LCP) 
_DPAT will start the working groups with stakeholders to develop 


predictive models in Spring 2015 
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EXAMPLE : Predictive Analytics draft model for M 


Speaker: Marie-Josee Dorion 
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DPAT - Next Steps 


Deliver on Projects 
al IN Project 
- TR Pilot 
+ FULL TR 
b} Express Entry -Phase 2 
ci eTA -phase 2 


Monitor and improve models {including QA and QC of performance of models} 


Continue Stakeholder engagement and outreach 


Consult and agree on model governance 


Develop IT solutions 


Speaker: Stephanie 
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Annex 1: Predictive Analytics App “0 


Cotlect Gute and Businesss Beguirements 
Conduct praleninary Bate Analysis 
Gevalon Predictive Models 
Validate Predicative Models. 


Conduct Statistical modeling of dak/complenity scoring 


Speaker : SK or MJ 


OF NOTE : THIS IS AN ITIRATIVE PROCESS, where we can go through steps one time, but we can also go between steps and come back if 
needed. 


The Predictive Analytics approach and methods is based on the CRISP-DM phases. 


CRISP-DM breaks the process of data mining into six major phases (and we have developed and agreed on a 7 step process) 


1- Business Understanding : This initial phase focuses on understanding the project objectives and requirements from a business 
perspective, and then converting this knowledge into a data mining problem definition, and a preliminary plan designed to achieve the 
objectives. 


2- Data Understanding: The data understanding phase starts with an initial data collection and proceeds with activities in order to get 
familiar with the data, to identify data quality problems, to discover first insights into the data, or to detect interesting subsets to form 
hypotheses for hidden information. 


3-Data Preparation: The data preparation phase covers all activities to construct the final dataset (data that will be fed into the 
modeling tool(s)) from the initial raw data. Data preparation tasks are likely to be performed multiple times, and not in any prescribed 
order. Tasks include table, record, and attribute selection as well as transformation and cleaning of data for modeling tools. 


4- Modeling: In this phase, various modeling techniques are selected and applied, and their parameters are calibrated to optimal values. 
Typically, there are several techniques for the same data mining problem type. Some techniques have specific requirements on the form 
of data. Therefore, stepping back to the data preparation phase is often needed. 


5- Evaluation: At this stage in the project you have built a model (or models) that appears to have high quality, from a data analysis 
perspective. Before proceeding to final deployment of the model, it is important to more thoroughly evaluate the model, and review the 
steps executed to construct the model, to be certain it properly achieves the business objectives. A key objective is to determine if there 
is some important business issue that has not been sufficiently considered. At the end of this phase, a decision on the use of the data 
mining results should be reached. 


6-Deployment: Creation of the model is generally not the end of the project. Even if the purpose of the model is to increase knowledge 
of the data, the knowledge gained will need to be organized and presented in a way that the customer can use it. Depending on the 
requirements, the deployment phase can be as simple as generating a report or as complex as implementing a repeatable data scoring 
(e.g. segment allocation) or data mining process. In many cases it will be the customer, not the data analyst, who will carry out the 
deployment steps. Even if the analyst deploys the model it is important for the customer to understand up front the actions which will 
need to be carried out in order to actually make use of the created models. 
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Key Steps to Success with Predictive Analytics 


Simplify the concept of predictive enalytics for business users and managers. 
Executive sponsorship for predictive analytics projects 
A consistent process for collaborating with the business stakeholders 
usiness outcomes for predictive analytics work 
Adequate fT support 
Proper communication of the results of predictive modeling to business stakeholders 
Having timely insights from predictive models in front of the business staff that can best use them 
À closed-loop feedback process for improving and updating predictive models. 
Executive endorsement of the results and for data driven decision making 


A center of excellence for date and predictive analytics 


Speaker : SK or MJ 
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e Solutions and Information Management Branch 

e Program Integrity Division, Operational Management and Coordination 

e International, Central Processing, and Domestic Regions 

e Strategic Projects Office 

e Data and Predictive Analytics Unit, Operational Performance Management Branch 
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Predictive Analytics 


Predictive analytics describes the process of discovering patterns in past behavior to predict the 
outcome of future events. It combines computer science, statistics and operations research to arive 
smarter decisions by extracting actionable insights from vast quantities of data. 


Timelines, Milestones, and Dependencies 


e Models for Automated triaging and auto-assignment of Express Entry applications for Fall 2015 for 
Phase 2 of modeling (working with real express entry data), for a roll out of a Predictive Analytics 
model in late spring 2016 


e Business rules for triaging of eTA applications developed in February 2015 (phase 1 of the eTA 
work}. 


Operational implementation of the models is critically dependent upon integrating the team’s work with 
that of SIMB, as data exchange with GCMS is crucial to apply the models to applications in progress. 


Level of Effort 


Moderate - 


A significant amount of work has 
been undertaken to acquire the necessary software tools and develop internal expertise. 


e Data and Predictive Analytics Unit, Operational Performance Management Branch 
Key Supporting Groups in CC 


e Solutions and Information Management Branch 

e Programs, Operational Management and Coordination 

e Program Integrity Division, Operational Management and Coordination 

e = International, Central Processing, and Domestic Regions 

e Strategic Projects Office 

e Performance Management Unit, Operational Performance Management Branch 
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Risk Tiering 

Risk tiering is a systematic approach to processing cases based on their likely level of complexity. When 
risk tiering, individual cases are assessed for their level of complexity using a checklist of individual triage 
criteria, which typically vary by line of business. Triage criteria are risk indicators which have been 
demonstrated to be useful in differentiating between aspects of complex and non-complex cases. 


Once a case’s level of complexity has been assessed it is then assigned to an appropriate decision 
maker for processing. 


ape 3 


Timelines, Milestones, and Dependencies 


Level of Effort 
- Development of effective risk tiering is dependent upon the development and validation of 
effective risk indicators and triage criteria, 


. Risk tiering rules must also be monitored and 
validated on a consistent basis to ensure that they work as designed, both when they are first 
implemented and as operational conditions evolve over time. 
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Options for a Predictive Analytics Pilot 


Operations Planning and Performance Branch 
November 23, 2016 
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Predictive Analytics Projects Underway 


Maturity Scale 


Field trial of China TRV model 
Ready for pilot 


Risk triage for In-Canada Spousal Inventory Reduction 


Risk triage models for Express Entry 


Risk indicators for Passport identity fraud in development 


Risk indicators for eTA 
Risk indicators for citizenship fraud 
Medical assessment targeting 


Biometric queries targeting Initial scoping 
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Progress has been made in designing IT infrastructure for 
deploying models 


Since Spring 2016, OPPB has been working with SIMB to design the IT infrastructure 
required to deploy a predictive model. 


The detailed business requirements were completed in September 2016. 
SIMB will soon begin designing the system. 


Once this work is completed, SIMB will have a blueprint of how to build and maintain the 
system. 


in the meantime, pilot projects can be used to test the performance of models in the real 
world. 


A-2018-09320-001217 


Two pilot projects already in progress 


China TRVs 


Family class spousal sponsorships 


«  InJune 2016, predictive models were used to perform a risk triage of the entire in-Canada 
spousal inventory. 


Each application was assigned a rating of high, medium or low risk, to assist officers when 
deciding how much scrutiny may be required. 


Among finalized applications since june, those that were identified as low risk had a refusal rate 
of just 4.7%, while high-risk cases had a refusal rate of 24.6% (overall refusal rate is 97). 
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Options for another pilot 


The following slides present seven options for an additional low-risk pilot. 
degree of readiness to proceed with a pilot in the near term. 


1. 


2: 
3 


4. 
5. Enhance risk triage for Federal Skilled Workers. 
6. Improve risk triage of Citizenship Grants. 


7. Improve targeting of medical assessments for TRs. 


Other future applications under consideration include: 
* Optimizing the management of the ATIP workload. 


* Enhanced client service through the Call Centre. 
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H 
Advantages 
* Streamlined processing with high accuracy. 


Model will have already been validated in 
the field. 


| Timeline: Early 2017-18. 


Immigration, Réfugiés 
et Citoyenneté Canada 
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Resource constraints at EDW mean that: 


+ OPPB would have to fund a consultant 
for them (850K). 


+ Project delays could occur. 


Costs: About $50K in consulting fees for EDW in 2016-17. OPPB has the money. 


| Next steps: 
_* Await results of field trial of the model. 


Work with CN, IN and EDW to develop the tool. 


Conduct additional field trials, if deemed necessary. 
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| 
Advantages 


* * Overlap with CN's existing business rules: 
CN's rules are very good; incremental j 
efficiency gains may not be large enough to | 


Low risk clientele which has been approved justify the edditional effort. 


recently, 


Simple process makes it easier for a model 
{e.g no supporting documents). 


| Timeline: Early 2017-18, 

Costs: Manageable within existing budgets. 

| Next steps: 

+ Assess overlap between predictive model recommendations and existing business rules. 


* Refine existing model, 
|* Conducta field trial, 
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RE e 


* Streamlined processing for a program with | * 
considerable volume. 


+ Low risk clientele. 

Uneven eligibility requirements: eligibility 
criteria and documentation requirements | 
vary by country, making it difficult to build a | 
general-purpose model. Might require l 
multiple models. 


+ 


Timeline: Early 2017-18. 
| Costs: Manageable within existing budgets. 


Next steps: 
_¢ Develop and test one or more models. 
Conduct a field trial. 
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s.16(2) 


| Advantages 


* Streamlined processing with high accuracy. 
Model simulations show promise. 


N/R 


+ Buy-in from fine of business: Unclear if this 
isa priority or if resources are available. 


| Timeline: Early 2017-18. 


Costs: Manageable within existing budgets. 
E Next steps: 
* Seek buy-in from line of business. 
:»* Refine model. 
Conduct a field trial. 
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5. Enhance risk triage for Federal Skilled Workers 


Advantages 


* Possibly some improvementin accuracy Express Entry keeps changing, making it 
relative to the current risk triage process. harder to develop an accurate model. 


+ Link to MYLP: FSW are an important source 
of projected growth in levels. 


| Timeline: Early 2017-18 

| Costs: Manageable within existing budgets. 

| Next steps: 

| * Assess the impact of recent changes to Express Entry on the existing model for FSW. 
Refine model. 

:* Conduct field trial. 
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6. Improve risk triage of Citizenship Grants 


Advantages Drawbacks / 
* Supports our response to Auditor * Data require cleaning: past applications are l 
General’s concerns. a combination of “legacy” and “new build”. | 


Efficiency gains: Streamlined processing for 
low-risk clients could help manage 


ressures resulting from shorter residence 
tes ents ‘eit cel. + Low rate of fraud detection in the past 
makes it difficult for a model to “learn” to 
Buy-in from the business: support from recognize fraud. 
CPPG, PIB, CMB, CN and DN, 


Timeline: Middle of 2017-18 
Costs: Manageable within existing budgets. 


| Next steps: 
i* Continue cleaning the data. 
‘+ Begin developing a model. 


je Conduct a field trial. 
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7. Improve targeting of medical assessments for TRs 


For TRs, the decision on whether a medical assessments is necessary hinges on regulations which 
may not be rooted in strong evidence. 


Improved targeting of heaith threats. * Resource constraints at Migration Health 


; NE Branch would push the start of this project 
Faster processing for low-risk clients. into 2017-18. 


Buy-in from line of business: Migration 


New subject area for OPPB means a 
Health Branch is eager to collaborate. 


steeper learning curve, a longer model 
development period, and greater 
uncertainty of results. 


imeline: By end of 2017-18. 


Costs: Likely manageable within existing budgets. 


| Next steps: 
Gain a deeper understanding of the medical assessment process. 
Gather available data. 
Develop and test a model. 

i+ Conduct a field trial. 
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Next steps after selecting an option 


* Work closely with affected branches, especially PIB, to: 
Refine the scope of the pilot. 
inform model development. 
Ensure program integrity safeguards and risk mitigation. 


Define ex post quality assurance activities and reporting. 


* Develop a governance framework for predictive models and risk indicators. 


* Ensure ongoing communications with other branches on plans and priorities. 
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Annexes 
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Value Proposition of Predictive Analytics 


* Risk management is built into the process by design. 


« Generates useful knowledge for program re-design in the longer term. 
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Difference Between Business Rules and Predictive Analytics 


While business rules and predictive analytics have some similarities, they are not 
one and the same. 


Business Rules Predictive analytics 
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Pros and Cons of Predictive Analytics Compared to 
Business Rules 


vantages 


Exhaustive. Models consider how all available 
information correlates with the outcome. Business 
rules are limited by our imagination. 


Granularity, Models create rules with many 
conditions that target very similar cases. 


Flexibility. Business rules that are hard-coded into a 
system cannot be updated regularly; models can be 
updated more easily. 


informative. Models use probability scores to 
specify each the degree of risk, rather than just a 
“pass” or “fail. 


Drawbacks 


* More complex. Predictive analytics requires 
specialized skills and software. 


+ Higher upfront cost. Models take more time and 
resources to create and deploy. However: 


~ {T infrastructure required for predictive models 

has large economies of scale and can be easily 

cute. es of business with smaller 
incremental coats. 


her volumes of low- or high 
generates efficiencies to 
‘ont costs over time. 
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Ravi: 


The purpose of this presentation is to provide an update on the development of 
Program Risk Indicators and associated triage criteria which will support 
the implementation of Express Entry 


This work in being done by OMC — Permanent Resident Program 


Delivery (PRPD), OMC — Program Integrity Division (PID) and OPMB Data 
and Predictive Analytics Team (DPAT) 
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Overview of CIC triage development 


Phased development of Express Entry triage 
criteria and workload distribution 

1. Phase 1~ Launch of Express Entry 

2. Phase 2 — Fall 2015 


Next Steps 


Ravi: See slide, no further details to add for speaking points. 
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Overview o 


David: 


A 
A-2018-09320-001234 


Ravi: 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


 Phased Approach 


Criteria to triage Federal Skilled Work (FSW) applications under 
Express Entry is being developed in phases to flexibly respond to the 
known and unknown demands of the new system: 


— Triage Phase 1:Launch of Express Entry (January 1%, 2015) 
“ “Manual” approach (similar to existing approach with paper applications) 
~ Triage Phase 2: Fall 2015 


+ Automated triage and scoring based on predictive analytical modelling layered with 
business rules and workload distribution models 


As the majority of Canadian Experience Class (CEC), Federal Skilled 
Trades Program (FSTP) and Provincial Nominee Program (PNP) 
applications are already processed in Canada, development of 
application triage instructions is not required for Triage Phase 1, but 
are being examined for Triage Phase 2. 


Canada 


More details on each phase will be provided in subsequent slides, this is just the overview 
slide. Please use bullets as speaking points. 


A 
A-2018-09320-001235 


à Immigration, Refugees Immigration, Réfugiés 
and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 


L'information divulguée en vertg dd BRINK s à l'information 
s.16(2) 


David: 


= 
A-2018-09320-001236 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


issued an ITA. 
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and workload 


Ravi : 
Point 1: MJ can jump in for tech related questions. 


Point 2: 
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Predictive Anal) Modelling: Steps 


Marie-Josée: 


Links with previous slide : Before we go in the results and the details, a few words on what this 
is that we do in predictive analytics. 


To achieve automated triaging and to develop and implement evidence-based & quantitative 
triage rules, OPMB-DPAT has worked with: 
-Partners and clients: 

-OPS partners, such as OPMB-PMU, OMC-PRPD, OMC-PID, IR, CPR, and SPO. 


-Historical data: 

FSW: 

Total number of valid records consist of 155,665 applications 

Total number of fields is 122 unique fields 

Dates covered (final decision dates) are from 2007-01-01 to 2008-02-27 


PN: 

Total number of valid records consist of 111,060 applications 

Total number of fields is 122 unique fields 

Dates covered (final decision dates) are fram 2007-01-02 to 2014-08-18 


CEC: 


Total number of valid records consist of 25,058 applications (GCMS only) 


Total number of fields is 122 unique fields 
Dates covered (final decisinn dates) are fram 2011-07-16 ta 2014-08-15 


Note: The goal is to “automate” risk tiering and auto-triaging 


= 
A-2018-09320-001239 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


s.16(1)(b) 
s.16(2) 


Triaging Rules for E 


Business rules were developed by the model for possible auto-triaging and 


auto-assignment. 
These rules were presented as If Then... statements that can be imposed 


on upcoming applications for triaging. 
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CEC dashboard was prepared by DMT and DPAT to test business rules 
provided by the model for possible auto-triaging and auto-assignment. 
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Accuracy of SPSS Triage Rules Outcome vs. GCMS 
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Scores and models are not everything: other rules in GCMS will be combined with the SPSS models 
scores so that triaging is automated and as accurate as possible. 

The placement of the business rules might change after we have come up with a Solution (IT) as this is 
just tentative for now. 


Complexity will be added in the future to the Predictive Models, in order to do complete complexity and 
risk-based triaging. 

Quality Assurance is a random selection of XX% of cases for QA, assigned to QA group. 

Workloads Distribution: Each bin can be assigned to officers according to operational requirements, expertise, levels, etc. 

Triage Model Validation: A random selection of cases from bin #1 and # 2 will be assigned to officers at superior levels to ensure that: 
Higher ranking officers remain exposed to all types of case. 

Senior officers can continuously validate the Triage model. 

Experienced officers can confirming that cases were effectively assigned to the right bin by the Triage model. 

Experienced field officers can detect errors in the Triage model. 


Experienced field officers can provide intelligence to centralised offices and HQ in order to improve and rafine the Triage model. 


Note: For each application that comes into GCMS, the Predictive Analytics Models will produce the 
probability that predicted FinDec will be approved or refused. For FSW, PN, CEC was used C5.0. 
algorithm to develop the models. Each of these LOB has different output for probability bins. 
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BERE 


Continue Express Entry Predictive Model validation ann business rules 


refinement until Fall 2015 forall lines of business under Express Entry 


Fraud Trend — Exercise in Fall 2015 once enough Express Entry Pata 


Marie-Josée/Ravi/David: 
Details: 
*It is anticipated to restart WG after receiving Express Entry data. 


* need to validate model and allocation, but any feedback soon in the 
process will be appreciated. 
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Additionally, some theoretical Indicators developed by PID include values that are not available 

in the source data. Therefore, the Program Risk Indicators developed for the FSW program 

were adapted to make use of the available data. A drawback to this approach is that the parts 

of the Risk Indicators that are not available in the data cannot be verified using statistical 

analysis and will have to be validated manually. To facilitate future automated triaging of cases 

based on the full set of Program Risk Indicators and predictive models, some characteristics of S:16(1)(b) 
officer-work and of applications will need to consistently be captured within dedicated fields in 
GCMS rather than in notes, although these changes may take some time to develop and 
implement. 


s.16(2) 


Various combinations of the Indicators were tested against historical data to determine the 
overall approval rate and the volume of applicants that the parameters captured. 


Use of this Indicator would result in the triage of an insignificant volume of FSW cases as non- 
complex and would not be useful for the purpose of capacity-based distribution 


While this process was effective in refining some of the Program Risk Indicators, the complete 
list of Indicators is not finalized and will continue to be reviewed as new/evidence-based 
information or predictors are determined resulting from fraud trends analysis and further 
testing/validation. 


DPAT has undertaken a detailed analytics exercise on FSW data to identify business rules and 
dependencies to model final decisions based partially on PID’s Program Risk Indicators. 
Indicator fields within GCMS were agreed upon by the Express Entry Data Analytics Sub- 
Woking Group.” 
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Next Steps 
OMC-PID will: 
. | | ; s.16(1)(b) 
* Support Permanent Resident Program Delivery (PRPD) in the drafting and further s.16(2) 


validating of file triage procedures for Express Entry to allow for the manual triage of files 
at launch 

Continue to support and validate DPAT Express Entry modeling work via ad hoc 
meetings and regular Express Entry Data Analytics Sub-Working Group meetings 
Continue working with DPAT in refining Program Risk Indicators and identifying new 
ones based on data and fraud trend analysis and further testing/validation of historical 
data 

Work with DPAT and PRPD on triage criteria / business rules for FSW and other 
Express Entry lines of business 

Support targeted program integrity and validation exercises on an ongoing basis 
following implementation of Express Entry PR triage criteria 


OPMB-DPAT will: 


Triage/Business Rules for FSW 

Try testing the FSW model on new data 

Produce final FSW modeling document 

Continue data analysis and modeling for PNP Program. 
Begin examining data for CEC program. 
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Annex: 


Revision of Program Eligibility Risk Indicators in FSW Programs 
s.16(1)(b) 


Background s.16(2) 


The FSW review/testing was administered to see if triage could be applied based on file 
complexity and capacity. This test would help indicate whether there is an optimal threshold to 
effectively triage applicant cases to where there is resource capacity to receive volumes. 


Methodology 


The methodology used focused on identifying non- complex cases as opposed to complex 
cases. Past experience with other business lines has also proven that non- complex indicators 
are more effective in defining and streamlining file distribution along the operational spectrum. 
By identifying non- complex cases all other cases that fall outside these criteria are considered 
complex. 


The sample was compiled on random basis. 
Data Information 
The data used for review/testing included the following characteristics: 


e Data was for the period of 2009 to 2014, as this was the most recent program 
information in GCMS 

e Total Sample size used: 246,087 for all FSW final decisions (in GCMS) 

e Total of 4 scenarios and sub-scenarios were tested. 


e Please note data was randomly compiled 
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Discussions with PID and DPAT 
September 27, 2016 
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Purpose of this presentation 


+ Present how we can best measure outcomes in this electronic 
world. 
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° Opportunity 


Protected È 


The Concept 
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e eTAs were originally meant to provide an advance screening 
mechanism for low-risk visa-exempt foreign nationals. 
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s.16(2) 


Protected B 


Protected B 
Measuring Outcomes 


e With GoC emphasis on Deliverology, measuring outcomes critical 
to demonstrating the success of an initiative. 
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Measuring Outcomes (A Practical Vision) 


Protected B 


Governance 


e Astable and nimble governance team will be key to positioning 
the project for success. 
e It will be responsible for: 
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Aba 27/09/2016 


Opportunities 


e Provide an opportunity to fully measure outcomes. 
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Deputy Minister adjointe déléguée 
Gperations Opérations 


Offawa KIA 111 


F-962821 
GCDOCS- 190290256 


PROTECTED A 
MEMORANDUM TO THE DEPUTY MINISTER 


ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING TOOL TO SUPPORT 
LITIGATION MANAGEMENT -PILOT PROJECT 


FOR APPROVAL 


SUMMARY 


e The purpose of this memorandum is to seek your approval to move forward with 
procurement activities to support a pilot, planned with the Department of Justice 
(DOJ), using an artificial intelligence/machine learning technology solution for 
immigration law. An artificial intelligence solution is sought to support legal 
research, analysing and predicting potential outcomes in litigation, and for trend 
analysis to support broader litigation strategy and program design. Artificial 
intelligence solutions would be used to support the work of officials of both the 
Department and the DOJ. 


e A Request for Information is being prepared. The targeted timeline for posting is the 
end of March. We are seeking specific information from industry regarding what 
technology is available or could be developed. This is an initial exploration with 
industry and what we learn will inform whether we move forward with a Request for 
Proposal or another approach to procurement. 


e We recommend that you approve this proposed initiative in order to move forward 
with this initial procurement step for an artificial intelligence/machine learning 
solution and related services for a litigation support pilot project by checking the ‘1 
concur box and signing this memorandum by March 20, 2018. 


Canada 
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s.21(1)(a) 
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s.21(1)(b) - 
PROTECTED A 


BACKGROUND: 


e Immigration, Refugees and Citizenship Canada (IRCC) and Employment and Social 
Development Canada (ESDC) are each partnering with the Department of Justice (DOJ) to 
explore the use of artificial intelligence and machine learning technologies and service options 
for separate pilots relevant to their work. IRCC’s pilot will be in relation to immigration law 
and for ESDC’s pilot will be in relation to the Government Employees Compensation Act. The 
Request for Information is seeking information for both pilots. 


e The objective is to support the legal and litigation work performed by DOJ counsels, litigators 
and IRCC officials with a view to adding valuable information and insights to be leveraged in 
the performance of this work, finding efficiencies and saving time that can then be spent on 
other tasks 


The Request for Information will explore the capability, viability 
and maturity of commercially available artificial intelligence and machine learning solutions 
that can: 

support case law and legal research: 

» predict litigation outcomes and assist in the development of le 
assessments: 

* facilitate trend analysis in litigation to inform litigation strategies, policy 
development and changes to operations: 

e explore the potential for an “Artificial Intelligence assist-tool” for IRCC 
administrative front-end decision makers: and 

ə reduce litigation costs across various business lines. 


gal advice/legal risk 


= 


© The Request for Information seeks to leverage industry expertise and obtain their perspective 
on the approach to these pilots including information in relation to the viable business models, 
contract considerations and hopefully resource and cost implications. 


e By way of background, in 2017 the DOJ had a contract with an industry start-up, Blue J Legal, 
for a feasibility study to determine whether an artificial intelligence and machine learning tool 
could be developed to answer discreet immigration law questions using this technology and 
data available. Blue J Legal has developed artificial/machine learning tools designed to be 
used by professional practitioners in the fields of tax and employment law. These tools can be 
used to predict the outcome of litigation in respect of specific legal questions on a given set of 
facts. These tools provide a percentage likelihood of the court’s ruling on the given question, 
together with a narrative analysis to support the conclusion that cites the key cases. The tool 
performs a real time analysis of the facts in light of every case decided on a given legal 
question. 


+ In June 2017, Blue J Legal presented their feasibility study to the Legal Issues Management 
Committee of IRCC. ` 
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PROTECTED À 


The Request for Information will allow industry vendors to provide further 
information on the scope of artificial intelligence and machine learning solutions that are 
available and will help inform the Department on whether to move forward with a tool limited 
to legal services/litigation support or one that will support administrative decision-making 
and/or possibly other functionality. 


CURRENT STATUS: 


e IRCC and the DOJ are working together to explore the development of artificial intelligence 
and machine learning solutions for immigration law with a view to launching a pilot of such a 
tool. 


e A Request for Information is being drafted as a first step towards the development of an 
artificial intelligence and machine learning solution for immigration law. 
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ESDC has joined the IRCC/DOJ draft 


Request for Information because they are also interested in developing artificial intelligence 
and machine learning tools for law. 


Deadlines 
are to be confirmed but the plan is to post the Request for Information towards the end of 
March 2018. 


CONSULTATIONS: 


_ For instance. IRCC has completed a pre-pilot in 
November 2017, testing automated decision-making of positive decisions, driven by predictive 
analytics, for the immigration offices in the China processing network’s Temporary Resident 
Visa line of business. The next phase will be a pilot in China in March 2018 with expansion to 
Temporary Resident Visa applications from India. later in 2018. 


+ Consultations with Public Services and Procurement Canada are continuing. 


+ A departmental working group has been established that includes Case Management Branch, 
Departmental Legal Services, Solutions and Information Management Branch and the 
Procurement group from the Administration, Security and Accommodation Branch. 


RESOURCE IMPLICATIONS: 


® It is difficult to estimate the financial costs of this initiative as thev are unclear at this time. 
Preliminary estimates point to a range of roughly solely for the 
development of a solution by industry to pilot a tool where a classifier is available for one 
IRCC business line (e.g. Pre-Removal Risk Assessments or Humanitarian and Compassionate 
applications). This preliminary rough estimate does not cover all internal costs such as 
planning, managing the project, implementation, training and maintenance. Discussions with 
Public Services and Procurement Canada are continuing regarding procurement activity costs. 
The purchase price, and the development and implementation costs are not known at this time 
and depend upon the solution. 


5 


+ Immediate costs for the procurement process are manageable. It is anticipated that the results of 
the Request for Information will allow the Department to estimate the costs for the pilot project 
with more precision. As the project progresses [RCC will be better positioned to proceed with 
detailed cost analyses to ascertain the investment needed to cover the total costs of all of the 
phases of the artificial intelligence and machine learning litigation project. 
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PROTECTED A 


e In terms of human resources, a combined total of one FTE in the Case Management Branch is 
working on the initial set-up phase (planning and initial procurement activities) relying on 
ongoing support and collaboration of departmental staff members who have been involved in 
this proposed initiative to date. The funding for this work is currently being absorbed at the 
branch level. As the project evolves, more resources will be needed. Case Management Branch 
will work in close collaboration with the Financial Management Division to monitor and 
ascertain specific needs. 


e IRCC is in the process of entering into a Memorandum of Understanding with Public Services 
and Procurement Canada to assign dedicated resources and expedite timelines for the project. 
Based on the financial implications of some other artificial intelligence initiatives being 
developed in the Department, costs to cover dedicated resources and procurement activities for 
Public Services and Procurement Canada could total around $125,000. 


e Financial Management Directorate has advised that, in the initial set-up phases (Le., costs for 
the Public Services and Procurement Canada Memorandum of Understanding, first-year FTEs, 
etc.), a strategy of attempting to manage costs through surpluses within Case Management 
Branch (and by extension, the Operations Sector) could be employed as a first step. Failing 
that, the Department would be able to secure another source of funds as the need arises. As the 
extent of ongoing costs becomes clearer, a longer-term financial strategy would be developed. 


COMMUNICATION IMPLICATION(S): 


e As this memo concerns procurement activities intended for internal use, communications 
implications are expected to be fairly minimal. Responsive key messages will be prepared as 
needed. 


RECOMMENDATION: 


e We recommend that you approve this proposed initiative in order to move forward with the 
Request for Information and related procurement activities for an artificial intelligence and 
machine learning solution and related services for litigation support by checking the ‘I concur’ 
box and signing this memorandum by March 20, 2018. 
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NEXT STEPS: 
e Continue the preparation of a procurement plan with Public Services and Procurement Canada. 
e Launch of the Request for Information to private industry. With information received in 
response to the Request for Information, the procurement plan would be updated and it is 


expected that a Request for Proposal or another procurement approach would follow. 


e Creation of a preliminary work plan and work flow process for the project with consideration 
as to which IRCC activities would be in scope for the pilot. 


e Monitoring project workload to ascertain if additional dedicated resources for this initiative in 
Case Management Branch and other areas are needed. 


de 


Dawn Edlund 


Ai concur 


(_] Ido not concur 


La | 


Marta Morgan 
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1 Background and Purpose of this Request for Information 
Objectives of the RFI 


Through this Request for Information (RFI), Immigration, Refugees and Citizenship Canada 
(IRCC), Employment and Social Development Canada (ESDC) and the Department of Justice 
(DOJ) are seeking to understand how each respective department could leverage the benefits of 
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artificial intelligence (AI)', machine learning (ML) and predictive analytics (PA) technologies to 
support decisions involving legal considerations, support the development of litigation strategies, 
and generate new insights and efficiencies in the delivery of legal services and litigation support to 
the Government of Canada. 


There are two separate pilot projects being explored. IRCC and the DOJ are partnering to develop 
an AI/ML powered solution for immigration law. ESDC and the ee are ona to ee an 


sedkial, AI/ML powered solutions leveraging data- driven oe are sought to assist Cand 
inform three activities: 


e legal research and development of legal advice/legal risk assessments; 
e prediction of outcomes in litigation; and 


e trend analysis in litigation. 


Information is also sought in relation to whether use of any AI/ML powered solutions could be 
expanded, in future, to users such as front-end decision makers. 


While IRCC and ESDC are pursuing separate projects, the two departments share the objectives of 
using this RFI to receive feedback, ideas and suggestions from industry respecting AI/ML 
technologies and solutions that will meet their needs. Information provided by industry will inform 
procurement(s) to support these pilot projects, should a procurement be pursued. 


Information provided in response to this RFI need not respond to both pilot projects and may be 
limited to responses in relation to an AI/ML powered solution for either immigration law or the 


GECA. 


The list of questions in section 5 are meant to allow industry to: 

a) share views and inform the direction that IRCC, ESDC and the DOJ may take with respect 
to AI/ML; 

b) comment on the risks associated with AI/ML powered solutions and mitigation strategies 
for reducing those risks; 

c) inform the parameters that IRCC, ESDC and the DOJ may consider in determining scope 
and when or if an AI application is appropriate for achieving desired business objectives; and 

d) provide ideas and suggestions as to how IRCC, ESDC and the DOJ could position their 
respective security and privacy frameworks in light of AI. 


The information provided will aid in increasing the understanding of the capability, viability and 
commercial availability of AI/ML technology solutions and services. The industry perspective on the 
approach, business and general requirements, cost of development, viable business models and related 
contract and intellectual property considerations are of particular interest. 


1 Recognizing that there is not a consistent agreement as to what constitutes artificial intelligence, 
including whether or not certain types of machine learning fall within the AI spectrum, for the 
purposes of this RFI respondents are asked to provide their responses considering a broad and 
inclusive approach to AI, including machine learning technologies. 
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Background 


The Government of Canada has been exploring data analytics for a number of years in an effort to 
understand developments in the field and its implications for federal departments and the legal 


industry. 


IRCC, ESDC with the DOJ are pursuing two separate pilot projects that will use AI/ML powered 
solutions leveraging data-driven insights to aid in: 

e research and development of legal advice/legal risk assessments; 

e prediction of outcomes in litigation; and 

e facilitating trend analysis in litigation. 


It is expected that the primary users of these two solutions would include the respective IRCC and 
ESDC departmental officials and DOJ counsel involved in litigation support. 


Information is also sought to inform potential expansion of AI/ML powered solutions to other 
uses/users in these domains such as front-end administrative decision makers. Such a solution 
could be used to assist and provide additional information for front-end decision making. 


While the business needs for each department, articulated in section 2, are focused on different sets 
of program activities, one set for immigration law and one set for GECA, the information obtained 
from this RFI will help inform each department’s broader AI/ML plans and further initiatives. 


2 Business Requirements 


This RFI seeks information in relation to two separate solutions, one for IRCC/DOJ and one for 
ESDC/DOJ. Each AI/ML solution is sought to help enhance: analytical capacity; speed, accuracy 
and effectiveness of legal research; speed, comprehensiveness and accuracy of answers to legal 
questions; prediction of litigation outcomes; and trend analysis in litigation. The additional objective 
is to streamline and find efficiencies in a number of different legal and business activities taking 


place within IRCC, ESDC and the DOJ. 


2.1 IRCC Decisions, Litigation and Trends 


IRCC is seeking a solution to assist in legal research, assist in the development of legal advice/legal 
risk assessments using predicted outcomes in litigation and to provide litigation trend analysis. There 
are many different areas of immigration law where IRCC officials make administrative decisions that 
are litigated. DO] counsel and paralegals, together with IRCC Litigation Analysts, review, respond to 
and manage the volume of litigation. 


Administrative decisions made at IRCC are subject to judicial review before the Federal Court. The 
DOJ has litigation counsel, DOJ Departmental Legal Services Unit RCC DLSU) litigation support 
counsel acting as in-house counsel to IRCC and paralegals who are variously involved from the 
point at which an application for leave for judicial review of an IRCC decision is filed through to its 
conclusion. Depending upon the complexity of the file, DOJ counsel, located either in regional 
offices across Canada or in Ottawa, may spend several hours per file researching, analyzing and 
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advising as part of the management of the litigation. Litigation Analysts at IRCC also spend several 
hours per file reviewing, analyzing, and liaising with program and policy areas and coordinating a 
departmental approach as the instructing client. This combined DOJ-IRCC effort includes time 
spent working together to determine the position(s) to be taken in the litigation, along with assessing 
implications and risk in relation to the litigation. Concerning the current environmental context, the 
DOJ] and IRCC conducts the legal research and legal analysis functions manually. On a preliminary 
basis, IRCC has areas of immigration law where it seeks to pilot AI/ML solutions: Pre-Removal 
Risk Assessment applications and Humanitarian & Compassionate consideration applications (both 
made from within Canada). The high-level functionality sought is as follows: 


e Allow users to conduct detailed research of case law relevant to a particular legal question 
and factor(s); 

e Use historical data to predict the likelihood of a case being successfully litigated; 

e Provide an analysis or narrative explaining which factors about a specific case are most 
relevant for predicting the likelihood of successful litigation; and 

e Identify/summarize the relevance of similar cases within the existing case law history. 


Optional functionality sought ts as follows: 


e Allow users to conduct exploratory data analysis of trends in immigration case law and the 
factors influencing these trends; and 

e A solution that could also be used by front-end administrative decision makers to predict 
litigation outcomes as a consideration in their decision-making process. 


A solution is sought to assist in the analysis of trends in immigration litigation/case law - a broad- 
based solution that can analyze large volumes of immigration litigation data (i.e., case law) to assist in 
the development of policy positions, program decisions and program guidance relevant to decision 
making and litigation would be ideal. For example, the capacity to analyze trends in case law, such as 
key facts that are influencing outcomes in litigation, would be valuable to IRCC and the DOJ. 


Users of the solution(s) 
The users of a possible AI solution would be IRCC Litigation Analysts, DO] Departmental Legal 
Services Unit counsel and regional DOJ litigators and paralegals who are involved in litigation 
management. With the responses to this RFI, IRCC will explore the possibility of whether or not the 
AI/ML powered solution(s) could also be used by front-end IRCC administrative decision-makers 
across the domestic and international networks to aid in their assessment of the merits of an 
application before decisions are finalized. This could entail separate client interfaces or functionality 
depending upon the user group. 


Available Data 
The data to be used by the AI/ML solution(s) would include published court decisions and general 
case factors, and, in future, possibly departmental applicant records (both structured and 
unstructured) contained in its system/database of record. 
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2.2 ESDC Decisions, Litigation and Trends: 


ESDC delivers a range of programs and services that affect Canadians throughout their lives such as 
providing Canada Pension Plan and Old Age Security pensions and related benefits, supporting 
unemployed workers, helping students finance their post-secondary education, and assisting parents 
raising young children. The Labour Program fosters safe, healthy, fair and inclusive work 
environments and cooperative workplace relations in the federal jurisdiction. 


ESDC is in the midst of a Service Transformation Plan which includes a number of service solutions 
that will leverage artificial intelligence and machine learning to deliver client-specific service insights 
and support. Likewise, ESDC’s Benefits Delivery Modernization initiative seeks to improve e- 
service and automation to streamline the processing of Canada Pension Plan, Old Age Security, and 
Employment Insurance claims, facilitate interactions with Canadians, and enable self-service for low 
complexity requests. 


ESDC’s Legal Services Unit provides legal services to support the programs, operations and key 
initiatives, including legislative and regulatory initiatives of ESDC, Labour Program, Service Canada, 
and Veterans Affairs Canada. The Legal Services Unit also provides litigation and litigation support 
services, which includes: 
e appearing before the Social Security Tribunal and Federal Courts of Canada on appeals and 
judicial reviews respecting the Canada Pension Plan, the Q “ve lef and the 
Tuublowment Insurance Ach 


SOCHITEY < 


e seeking settlement recoveries related to the Government Employee Compensation Act program, 
managed by the Labour Program. 


It is within this context that ESDC is pursuing information on AI/ML solutions which support early 
legal risk assessment, informs litigation strategy, and the identification of potential efficiencies in the 
delivery of legal services. 


The high-level functionality sought ts as follows: 

e Allowing users to conduct detailed research of case law relevant to a particular legal question 
and factor(s); 

e Determining the probability that a given fact scenario meets the threshold for benefit 
eligibility, or is a viable claim (e.g. likely to result in a settlement); 

e Identifying what information may be missing in order to meet this threshold; 

e Identifying the dollar range of the settlement that could successfully be negotiated; and 

e Identifying/summarizing the relevance of similar cases within the existing case law history. 


Should there be suitable tools available, ESDC would likely explore opportunities to pilot a solution 
for a select program area, in order to assess the potential for scaling the solution in other contexts. 
ESDC Potential Use Case: Government Employee Compensation Act (GECA) program 

When an employee of the public service of Canada suffers an injury resulting from an accident or an 


occupational disease i in the course of their employment, they are eligible for the benefits provided by 
g (GECA) (http://laws-lois.justice.gc.ca/eng/acts/G-5/). 
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Where a third party is liable for the injury, the Labour Program seeks to recover the Crown’s costs 
from that third party. Once assigned a case by the Labour Program, the Legal Services Unit leads the 
early assessment of claim viability, and leads the negotiation of settlements with the third party. 
Viable claims for which ESDC has not been able to negotiate a settlement, and higher value claims 
that have insufficient information to assess before the Statute of Limitation date, are sent to the 
DOJ for litigation. DOJ counsel, located either in regional offices across Canada or in Ottawa, then 
assume cattiage of the file, and may spend several hours per file researching, analyzing and advising 
as part of the management of the litigation. 


Available Data 
The data (structured and unstructured) to be used by the AI/ML solution and model includes 
published court decisions, an administrative GECA database, and additional administrative records. 


Users of the solution(s) 
Dispute resolution advisors, paralegals, and counsel within the ESDC Legal Services Unit would use 
this solution as a research aid in developing their strategy in settlement negotiations, as well as a 
resource in the determination of which claims to send to DOJ counsel for litigation. 


3 Instructions to Respondents 


Nature of request for information 


This is not a bid solicitation. This RFI will not result in the award of any contract. As a result, 
potential respondents of any goods or services described in this RFI should not reserve stock or 
facilities, nor allocate resources as a result of any information contained in this RFI, nor will this RFI 
result in the creation of any source list. Therefore, whether or not any potential respondent responds 
to this RFI will not preclude that respondent from participating in any future procurement. Also, the 
procurement of any of the goods or services described in this RFI will not necessarily follow this 
RFI. This RFT is simply intended to solicit feedback from industry with respect to the matters 
described in this RFI. IRCC, ESDC and the DOJ are interested in technology and solutions that 
would enhance the efficiency and effectiveness of the litigation management and front-end decision- 
making processes. 


Format of Responses Requested 


a) Format: Respondents are requested to submit one soft copy of their response in Portable 
Document Format (PDP). 


b) Cover Page: If the response includes multiple volumes, respondents are requested to 
indicate on the cover page of each volume the title of the response, the solicitation number, 
the volume number and the full legal name of the respondent. 


c) Title Page: The first page of each volume of the response, after the cover page, should be 
the title page, which should contain the: 
a. title of the respondent’s response and the volume number; 
b. name and address of the respondent; 
c. name, address and telephone number of the respondent’s contact; 
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d. date; and 
e. RFI number. 


d) Part A should include the Proposed Solution with references to the Business 
Requirements as depicted in Section 2 and the General Requirements in Section 4. 


e) Part B should include answers and feedback to the questions listed in Section 5. 
Please identify which department/pilot project the responses relate to. 


f) Numbering System: Each question has its own unique number. It is a sequential number 
prefixed with a “Q”. Respondents are requested to prepare their response using the 
numbering system corresponding to the one in this RFT, and to repeat the question prior to 
their response for ease of reviewer reference. All references to descriptive material, technical 
manuals and any brochures included as part of the response should be clear both in the 
citation and on the referenced document. All should be referenced accordingly. 


Response Costs 
The Government of Canada will not reimburse any respondent for expenses incurred in responding 
to this RFT. 


Treatment of Responses 

a) Use of Responses: Responses will not be formally evaluated. However, the responses received 
may be used by the Government of Canada to develop or modify procurement strategies or any 
draft documents contained in this RFI. The Government of Canada will review all responses 
received by the RFI closing date. The Government of Canada may, in its discretion, review 
responses received after the RFI closing date. 


b) Review Team: A review team composed of representatives of IRCC, ESDC and the DOJ will 
review the responses. The Government of Canada reserves the right to hire any independent 
consultant, or use any government resources that it considers necessary to review any response. 
Not all members of the review team will necessarily review all responses. 


c) Confidentiality: Respondents should mark any portions of their response that they consider 
proprietary or confidential. The Government of Canada will handle the responses in accordance 
with the Access to Information Act. 


d) Pre-Submission Industry Day Sessions: The Government of Canada may, in its discretion, 
host one or two industry day sessions for the purpose of explaining its requirements and to 
allow industry to ask questions and seek clarifications. 


e) Post-Submission Review Meetings: The Government of Canada may, in its discretion, hold a 
single Post-Submission Review Meeting with all interested vendors or request individual Post- 
Submission Review Meetings with selected respondents to provide clarity on information 
provided, or to invite a presentation about some or all of the proposed solution. If required, 
these will be held at the most appropriate location, to be determined at a later date. The intent of 
these meetings will be to provide an opportunity for a face-to-face discussion with respondents. 
Although respondents may request a meeting, and their request will be considered, the 
Government of Canada will determine whether or not it requires additional information from 
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any given respondent and will schedule meetings accordingly. All such requests, by respondents, 
should be forwarded to the Contracting Authority identified herein. Note that a maximum of 
two (2) hours will be set aside for any meetings with respondents. 


Enquiries 

Because this is not a bid solicitation, the Government of Canada will not necessarily respond to 
enquiries in writing or by circulating answers to all potential respondents. However, respondents 
with questions regarding this RFI may direct their enquiries to the Contracting Authority identified 
herein. 


Submission of Responses 
a) Respondents should send responses electronically via e-mail to the Contracting Authority's e- 
mail address identified herein by the date specified on the front page of the RFI. 


b) All requested information 1s to be provided to the Contracting Authority on or before the 
closing date of the RFI. 


Contracting Authority 
The Contracting Authority for this RFI 1s: 


Contracting Authority: Heather Ferrier 
E-mail Address: heather. ferrier@tpsgc-pwesc.gc.ca 
Telephone: 613-720-7986 


4 General Requirements 


Availability requirements 

The solutions may be made available to officials of IRCC, ESDC and the DOJ nationally (across 
Canada). The number of users will vary between activities. The user base may be extended to IRCC 
officials stationed overseas in possible future deployments of the solution(s). 


Language 
The Qf | 
instruments be made available in both English and French to government officers and employees. 
Consequently, the solution will require à bilingual user interface. Training documentation and user 
manuals will also need to be made available in both English and French. A desirable feature would 
be for the users to be able to toggle on-the-fly between English and French within the solution. 


ang Id RSC. 1085, c. 31 (4th Supp.) requires that regularly and widely used work 


LS RAC APS 


Support 
The respondent should indicate their organization’s approach to the management of incidents and 
problems and illustrate how issues are escalated and resolved. 


Maintenance 

The respondent must be prepared to illustrate their problem/change management procedures for 
ongoing updates to the solution(s), their maintenance procedures for the solution(s) and explain 
their approach concerning emergency fixes and problem escalation procedures and resolutions. ‘The 
frequency of solution updates/releases and the downtime, if any, associated with updates/releases to 
the solution would be helpful to know. 
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Training 
The respondent must provide à training strategy and approach to the creation and dissemination of 
user education material. 


Security 

The solution(s) must secure solicitor-client privileged information and data and must achieve a 
minimum of a Protected B level of security. Protected B is the federal level of security that applies to 
information and assets that, if compromised, could cause serious injury to an individual, 
organization or government. 


The respondent must be aware that, at this time, IRCC, ESDC and the DOJ assume that the security 
profile for Protected B information will be mid-integrity and mid-availability as per the Government 
of Canada’s FTSG-33 policy and procedures (as contained at the following link: bttps:/ /www.cse~ 

‘publication /its 


cstge.ca/en/ 30-33). 


It should be noted that for user access controls, the solution(s) need(s) to accommodate users from 
multiple departments IRCC/DOJ and ESDC/DO} for the respective solutions. 


It is desirable to implement Single Sign-on (SSO) using services such as Active Directory and other 
Public Key Infrastructure solutions to allow for remote access to Government of Canada networks. 


Access rights can or will be set for the DOJ, IRCC and ESDC or third parties as required. 


Reporting capability 

The respondent must describe the reporting capabilities of the solution(s) and any reports and data 
visualization provided with the solution(s) (e.g., ad hoc reporting, canned reports, visualization and 
drill down capabilities for analyzing algorithm decisions, etc.). 


5 List of Questions 


Company Overview 
1. Please provide a short description of your firm, its facilities and locations and the types of 
products and services it provides. 
i. In which country/countries are your facilities located? 


i. In which country/countries does your firm conduct business? 


ii. How long has your firm been in business? 
iv. Has your firm previously received security clearance to work with the Government 
of Canada? 


2. Have you provided AI/ML solutions or developed prototypes for public sector or private 
sector organizations in Canada or elsewhere? If so, what were the objectives and the high- 
level functionality of the solution(s), and what are some lessons learned from the 
experience(s)? 
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The Solution 


3. Please provide detailed information about a potential solution, including a description of: 
i, How the general business requirements could be addressed by the solution; 
i. Significant gaps in the identified requirements or how they could be improved; 
ii. The nature of the analysis and the supporting information the solution will provide 
to users for the three activities identified being research, prediction and trend 


analysis; 

tv. Whether or not the legal questions to be answered by the solution need to be 
discrete and fact-based; 

v. Conceptually, how the solution would take the facts of an individual case and arrive 


at a predicted outcome of the litigation; 
vi How the solution would be adapted to address changes in the law (case law or 
legislation); 
vit. How the algorithm or neural network is trained; 
viii. The volume or nature of training data (e.g., applications /claims/facta/judicial 
decisions) that would be required to allow for accurate predictions of legal outcomes 
and why that volume is needed. 


Viewpoint on AI/ML Technology, Maturity & Limitations 


4. What challenges do you foresee in developing and implementing AI/ML solutions and what 
solutions exist to overcome those challenges? What are the unique considerations in the 
government setting? 

i. How would industry address the challenge of demands to make AI/ML models 
transparent to ensure that the predicted outcomes can be reviewed and the rationale 
understood (e.g., which factors were the most important in influencing the predicted 
outcome, and how was the predicted outcome developed)? What would be the 
consequences in this context of releasing such information? 

ii. What are the pitfalls, weaknesses and critical dependencies in AI solutions for the 
identified requirements? 

iii. | Flow can AI/ML models be developed to ensure biases or potential biases are not 
introduced? How are biases detected? 


5. How would you characterize and define the different types of AI/ML technologies? 


6. What solutions do you consider to be mature, developing or in the early stages of 
implementation that are appropriate for the identified requirements? Are solutions available 
to meet the identified requirements or could solutions be customised or configured to meet 
these requirements? Should solutions not be available to meet identified requirements could 
a solution(s) be developed with available AI/ML technology? Please explain. 


7. Whether a Commercially Off The Shelf (COTS) products exists or a solution needs to be 
developed to meet identified requirements: 
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i. What level of client capacity and involvement is required to develop a solution or to 
support the use of COTS (including customization and configuration) for AI/ML 
solutions? 

ii. Would any solution developed or COTS products allow for AI/ML models to be 
developed, re-trained or adjusted as required for more than one area of the law or to 
answer more than one legal question? That is, how reusable/ transferable is the 
technology such that a solution could be adapted for use for multiple areas of the law 
or to answer multiple legal questions? 

iti. Is it possible that a solution could accommodate requirements and users from 
different departments IRCC/DOJ and ESDC/DO}] for the respective pilot 
projects) or are separate solutions required to accommodate their respective 
requirements and users? 

iv. Are there design considerations in an AI/ML solution that would allow it to have the 
widest range of potential applications/uses beyond the identified requirements? If 
yes what are the major design considerations? What are the key elements to consider 
in terms of such reusability and transferability? 


Approach to Developing and Implementing AI/ML Solutions 


8. What are the major considerations and critical components that need to be taken into 
account with respect to developing and operationalizing AI/ML solutions? For example: 

i. Are there complementary software or hardware dependencies that need to be in 
place for an AI/ML model to function correctly? 

ii, What interoperability capacity would your solutions have with other commonly used 
products (such as Microsoft SharePoint, OpenText document management, 
Customer Relationship Management (CRM), case management solutions, Statistical 
Package for the Social Sciences (SPSS), SAS, Oracle, etc.)? 

iii. Are there any special CPU, memory, storage, network or database requirements? 

tv. How is OS, server patching and third party middleware upgrades handled? (i.e., 
schedule, frequency, on demand)? 


9. What are the industry’s standards, best practices or measures that can be used to assess the 
efficiency, accuracy, reliability and performance of such AI /ML solutions? 
Í; Explain how efficiency, accuracy and reliability of such solutions would be ensured 
and measured? Please explain including in relation to the solution’s analytical and 
predictive capabilities? 


ii. Can you measure accuracy of predictions of outcomes/trends such as by a margin of 
error? If yes, what is the margin of error? 
iii. | Can you measure accuracy of the solution’s other outputs (information/analysis) 


such as by margin of error? If yes, what is the margin of error? 

tv. What is the performance of such solutions (e.g., response time for a 
research/prediction of outcome/trend analysis including the supporting information 
provided by the solution)? 
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10. Please advise of any other industry standards or best practices that should be followed for 
the development or deployment of AI/ML solutions that were not already addressed in your 
previous responses. 


11. With an AI/ML solution development and deployment, what is the role of clients? For 
example: 

i. Can client users train and re-train the AI/ML models, or would this require 

additional customization, support or permissions? 

Can clients configure data, rules, algorithms and fine-tune decisions? If yes how? 

iii. | Can data/rule configuration be done by the client in production or does it need 
back-end technical support/release management? 

iv. Are any specific programming and scripting skills/training required or additional 
professional services required? 

v. How can users review the algorithm decisions? 


2 pe 


Training & Support 


12. Please provide an overview of the training and support services available, including 
considerations for implementing the solution across Canada, potentially across different 
governement departments and possibly overseas in future iterations. 


Additional Technical Considerations 


13. Please explain: 


i the languages and/or character sets supported for the import, export and manual 
entry of data into the solution; 

ii. any dependencies on third party software components necessary to deploy the 
solution; 


ii. details related to the support of any third party components (e.g., source of support, 
method of delivery, maintenance, etc.); 


iv. integration points between requirements, where applicable, and any value-added 
software products /solutions; 

v. the level of Government of Canada IT resources required to support the solution 
(e.g., hardware, software, etc.); 

vi How many minor and major releases are planned in a year? Are there solution 


outages associated with the different types of releases? If there are solution outages, 
what is the average time of the outage based on release type? 
14. What are the various hosting options available for the proposed solution(s) that allow for 
data to reside and remain within Canada (e.g., Software as a Service (SAAS), Infrastructure as 
a Service (ITAAS), Platform as a Service (PAAS), on-premise instances, etc.)? 


1. For each option, please describe the security architecture and how it meets the 


The Rs Board Secretariat u (TBS) Cloud o Strategy is also included for 
reference: | 
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u. If cloud-based, how is tenancy security implemented? 

iit. Describe the business continuity plan to address unforeseen circumstances. 

tv. Please identify any other hosting considerations that should be taken into account by 
the Government of Canada. It should be noted that any off-premises hosting 
solution must meet the data sovereignty requirements for data to be hosted in 


Canada. 


15. What would be the expected length of time required to provide a solution that meets the 
aforementioned requirements? 


User Accessibility & Usability 
16. How does the potential solution address the following: 


i. What mechanisms are in place to meet Web Content Accessibility Guidelines 
(WCAG)? 
u. How are input data sets entered into the system (e.g., manual data entry, spreadsheet 
impotts, graphical)? 
ui. How are decisions reviewed (e.g., lists, graphical, reports)? 


Data 


17. What data sources would you recommend for use to develop such a solution to meet the 
identified requirements i.e. for the three activities identified being research, prediction and 
trend analysis? 


18. How can industry leverage and integrate public information repositories in AI/ML models? 
How can information repositories internal to government departments be integrated? 
i How would you solve the challenge of working with and analyzing large volumes of 
unstructured data? 


19. Should IRCC or DOJ databases be required, what is involved in leveraging or integrating an 
AI solution with existing databases and what data requirements /inputs would need to be 


developed for an AI solution, whether it includes internal data and/or publicly-available 
data? 


20. Are industry data standards used for the specification of inputs and decisions? 


21. Are industry-standard AI/ML algorithms used (e.g., Bayesian, Decision Trees, Neural 
Networks, Linear Regression)? 


Privacy & Protection of Information 


22. How does industry ensure that privacy and confidentiality are protected when applying 
AI/ML technologies to personal information holdings? 
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i. What best practices can be used to address privacy concerns and the requirements in 
the Privacy Act AE lois.i ne: gc. E ea Personal Information 


23. When analyzing sensitive data in an AI/ML model, how is confidentiality ensured for 
information that must be protected (e.g., solicitor-client privileged information)? 


24. How does the potential solution address IT security? 


i. Application: What mechanisms/processes are in place 
a) To prevent unauthorized access or data integrity compromise? 
b) For logging and auditing user events, rule / algorithm changes and AI 
algorithm decisions? 
c) To handle access control and at what level of granularity (e.g., field level, 
case level, decision level)? 


il. Information Management: What mechanisms/processes are in place: 
a) To protect the input data? 
b) For data retention and disposition? 
c) To package and transfer data back to the Government of Canada if the 
solution is discontinued? 
d) For reporting security incidents and violations? 
e) For disaster recovery and business continuity? 


Intellectual Property (IP) 


25. Please describe the options for the intellectual property rights, taking into account the viable 
business models for the solutions, such as vendor-owned and Government of Canada- 
owned. 


Cost Model 


26. Please describe the options for the pricing model for a vendor-owned solution (e.g., 
perpetual licence, subscription-based licence, user licence, device/CPU/server licence, 
entity/enterprise licence, other model)? What is the estimated cost to develop and maintain 
such products and services? 


27. Where a Government of Canada-owned solution is developed because a vendor-owned 
solution is not viable, what would be the estimated cost for the products and services? What 
is the estimated cost to develop and maintain such a solution? 
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" setting up tools (e.g., risk factor model, predictive analytics pilots) 


N/R 


o Maintaining processing times despite increasing volumes (i.e. improvements to TR processing, predictive 
analvtics) 
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e Meeting long-term processing objectives 
depends on leveraging innovation. Analytics 
technology can facilitate reduced 
investment in physical infrastructure, and 
reallocation of resources to focus on 
complex cases. 
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s.16(1)(b) 


s.16(2) 


Sector Focus: Risk 


Processing Times 


Predictive 
Analytics to 
Improve 
Efficiency 


e It will be increasingly difficult to 
manage growing volumes (e.g. TR) if 
predictive analytics is not implemented 
as a risk management tool to improve 
processing efficiency. 


A-2018-09320-001293 


Sector Focus: Client Service 


Immigration, Refugees Immigration, Réfugiés 
| + | and Citizenship Canada et Citoyenneté Canada 
Information disclosed under the Access to Information Act 
L'information divulguée en vertu de la loi sur l'accès à l'information 


Secret 


Client Support 
Centre 
Modernization 


User-centric Design and Delivery 


* Client Support Centre (CSC) quality 
assurance program revised to measure 
excellence in alignment with IRCC Service 
Excellence Action plan priorities. 

* Improvements to respond to the 
Modernization of Client Service Delivery 
Report tabled by the Standing Committee 
on Citizenship and Immigration in March 
2017. 

e Limited ability to resolve service issues in 
the near-term given CSC is currently 
operating at full capacity, with relocation 
planned for spring 2018. 


* Client service excellence initiatives will 
include increased accessibility for clients, use 
of analytics to predict client behaviour, and 
soft skills training for staff. 

* Pilot at CSC to engage staff in providing 
excellent client experience. 

e Timely collaboration among key partners 
(CPPG, DN, LDD, Communications) is key. 
*Implement initiatives from the Citizenship 
Design Challenge. 


N/R 
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Secret 


Workload Management and Employee Wellness: 
Employee change fatigue and ‘burn-out’ continue to 
be a risk due to frequent changes to organizational 
structure, legislation, and priorities. Measures to 
address include training and proactive change 
management and will be essential through the 
transition to a predictive analytics driven 
organization. 
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Predictive Analytics N/R 


* Further predictive models will be developed for China and pilot scoping is planned for India TR lines of OPP 


business 
* Pending a successful pilot for Citizenship, a model will be implemented for risk triage OPP 


* Pending a successful pilot for Family Class, risk indicators will be implemented for risk triage in 2017— 
2018, with expansion in 2018-2019 OPP 
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2015-2016 P12 / 2016-2017 Plan 


Adjustments for Operations Sector 
May 2016 


& 
Citizenship and Citoyenneté et d; ` 
immigration Canada immigration Canada a a 
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N/R 
A data mining project to identify fraud as well as work on predictive analytics indicators and models has been initiated to support Global Affairs 
Canada on trade and commerce with the United States. 
N/R 
N/R 
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Secret 
2016-2017 Risks Operations Sector will Mitigate 
Risk A — Failure to fully deliver on priorities and core objectives due to extensive change agenda 
The Operations Sector is undertaking and planning the following key actions in 2016-2017 to mitigate Corporate Risk ‘A’: 
N/R 

° Predictive Analytics Models development: (i) Predictive Analytics Models will be finalized and presented to Working Groups (International Experience 

Class, Visitor Record-Extension, Student Permits, Temporary Foreign Worker, Temporary Resident Visa); (ii) Operations Performance Management 

Branch will provide funding to Solutions and Information Management Branch to design and cost the IT infrastructure required for model 

deployment; and (iii) work will continue on risk indicators for Electronic Travel Authorization Expansion countries. 

N/R 
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PATRICK’S PART 


How is this item different from most Policy Committee items? 


e There’s no deck. it’s about a paper. We want you to read the paper. 


Where did this project come from? 


e Alittle history of the team — e.g. IRPA Part 4.1, other work on innovation 


e Inthe summer of 2017, SPP Branch’s Digital Policy team developed and consulted a series of issue papers 
relating to the future of e-tools and automation: topics like paperless applications and authorizations, the 
elimination of place-based application requirements, procedural fairness for clients, oversight of 
electronic systems 


e Inthe fall, TBS developed and widely distributed a White Paper on Responsible Artificial Intelligence in the 
Government of Canada . 


e In December, SPP presented to ExCom an update on our work to ensure that the Department’s policy and 
legal authorities support broader use of technology across all program lines 


+ As part of our forward plan, SPP Branch committed to developing a white paper in 2018 to help the 
Department think through the wide-ranging implications of technological innovation 


e Some broader Got context: TBS Standard and GAC development of a Canadian position 


What do we hope to achieve with this paper? 


è Boost the Department’s digital literacy 


+ Give people a common lexicon and framework for understanding the specific technologies that are 
relevant to IRCC’s operations, as well as a basis for weighing the risks and benefits associated with each; 


è Sensitize people to risks that may not be apparent at first glance 
è Provide interim guidance to program designers, policy analysts and senior managers; and 


+ Ultimately, kick-start a larger effort to ensure that as the Department continues to use technology in new 
ways, we ensure fairness and transparency for applicants, we mitigate new and unexpected risks, and we 
realize the full potential of transformative tools. 


What is this paper NOT trying to do? 


ə Outline a technology grand strategy for the Department 
e Catalogue all tech-centric projects at IRCC 


e Prescribe specific policy or program design choices 
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GREGG’S PART 


ll start by talking about who the paper is meant for and what it covers. 


e The paper is meant for: 
1. program designers — business owners, operational project leads 
2. policy analysts or legal counsel 
3. senior managers 


Less so, but still: those who work in IT, information management, HR, ATIP 


e Our paper builds on the TBS white paper, and focuses on IRCC’s specific set of challenges and opportunities. 


e |t includes a discussion of artificial intelligence, but casts a wider net. 


We knew we had to put some limits on the discussion. You can’t write a white paper about “new 
technology.” 


After some spirited debate, we settled on the following four tests to determine what technologies are in 
scope for a discussion about digital transformation at IRCC: 


1. Does the technology fall within the definitions of data analytics, automation or artificial 
intelligence? 


2. Do applications of the technology relate to outward-facing activities that are governed by the 
Department's legal framework? i.e. Do they relate to activities that ultimately affect external 
clientele, as opposed to strictly internal business such as HR and financial management? 


3. Beyond achieving efficiencies, could the technology alter decision-making processes or otherwise 
affect the fortunes of IRCC’s clients? i.e. Could technological change have unintended 
consequences or cause differential impacts on applicants, resulting in new “winners” and “losers”? 


4, Are applications of the technology proximate enough? Is piloting or implementation conceivable 
within the next three years? 


If people at this table are answering yes to these questions, we would really like to work with you to help 
you sort out the ethical, policy and legal issues. 


With the scope established, the paper then goes on to break down the issues in three core sections: 
(1) Potential benefits of these new technologies 
(2) Risk considerations (which | would call “pitfalls”) 


(3) Guiding principles and some practical recommendations to guide work that is in its early stages 
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The advantages of digital transformation are, by and large, well understood throughout the Department. 
They include: 


s.21(1)(a) 
% Reducing application backlogs and processing times 


% Freeing up officers to conduct more value-added activities, such as quality assurance or in-person 
interactions with clients 


“* Strengthening program integrity 


+, 
2G 


* Improving client service with electronic tools that can work around the clock, with greater 
efficiency, and in ways that clients find inherently more convenient 


** Making sense of the flood of client feedback about policy and program changes, thanks to text 
analytics that can deliver powerful sentiment analysis 


œ There are others 


The pitfalls of digital transformation are perhaps less well known. This is the real meat of the paper. 
Some of the big risks include: 


% Innovation could outpace our legal and policy frameworks. 


+ Anew part 4.1 of IRPA provides broad Ministerial authorities for the use of technology in 


immigration lines of business, but other language throughout the Act and its regulations reflects 
a presumption of physical documents and paper applications. 


** The Department’s existing privacy framework will almost certainly be encroached upon. 


e Data-driven technologies are, in a sense, fundamentally at odds with privacy aims. Data analytics 
and machine learning are all about feeding the machine as much data as possible. Conventional 


privacy wisdom is all about limiting the use of information to the original, narrow purpose for 
which it was provided. 


» Data needs drive increased information sharing between IRCC programs and with partners, and 
we will need to ensure appropriate legal authorities to mitigate privacy risks. 


ə | know that IRCC’s privacy experts are already thinking about things like 


o Revising PIAs as we incorporate advanced analytics and automation 
o “Doubling down” on Privacy by Design principles so that they become a default mindset 
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Introducing automated decision-support systems into our administrative decision-making processes 
could cause clients to identify a lack of fairness and transparency, and IRCC could inadvertently 
contravene principles of administrative justice. 


There is lots in this section, and | won’t go through it all. But here we are talking about things like: 
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So there are some serious risks....but we didn’t want to leave the reader with a sense of trepidation 
about innovation. A lot of thinking has been done in this area, and there are some nuggets of 
wisdom at the back end of the paper. (ld like to just highlight a few of them.) 


The paper mainly proposes guidelines for the responsible use of new technologies in administrative 
decision-making and client service. Among them: 


“+ Let computers focus on their strengths — reliably analyzing large volumes of data and considering 
millions of possibilities without fatigue — and let humans focus on theirs: intuition, creativity, 
empathy, social skills, shaping a larger strategy. Machines are best at choosing answers, humans are 
best at choosing questions. 

% When determining whether a task or step in a decision-making process could be shifted from 
humans to machines, program designers should ask themselves, “Is this a situation in which 
reasonable minds may differ?” If yes, automation would seem ill-advised. 

% Data will always bear the marks of its history. In using data to train a system to make equitable 
recommendations or decisions, we must be fully aware of the workings of this history. 

% Ina future business model centred on analytics and automation, clients will still have the right to an 

explanation of IRCC’s decisions, and this explanation need not jeopardize program integrity. There 

are ways to explain decisions made by algorithmic systems without disclosing the source code and 
allowing people to game the system. 


“* We must take ownership for the successes and failures of electronic systems. Tools like chatbots 
should be treated as agents of the organization and held to the same standards as human agents. 


(Since | couldn’t help myself} the paper ends with some other recommendations that relate to creating 
the conditions for a successful digital transformation. These include: 


*e Data quality is the single greatest determinant of success in projects involving data analytics. 
Prioritizing analytics means prioritizing the breadth, depth and reliability of our data sets. 


* Increase our store of in-house expertise. Recruit people with new skill sets and invest in education 
for existing personnel. Data literacy is important for all. 

% Build resilience in preparation for major disruptions, be they lawsuits, cyber-attacks or other. 

% Novel and complex as it is, the technology is but one piece of the puzzle. Removing barriers to 


adoption, changing long-standing business practices and shifting our organizational culture could be 
greater challenges. 


% Get the governance right. Remove artificial barriers that keep the right people from working 
together. 

% The success of innovative tools lies not in the technology per se, but in our ability to align it with our 
business models. (Human + machine > hardest part is the +) 


I’ve talked enough. We’d like to hear from you — in the 
time we have left, and in the weeks and months to come. 
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QUESTIONS FOR THE TABLE: 


Given that IRCC has numerous experiments in the works, TBS is preparing a regime to vet and oversee 
algorithmic systems, and GAC is preparing a Canadian position on Al for international relations, what 
should SPP do next? 


ə Create a tool, like a checklist, to help people assess and manage projects? 


e Write a Departmental policy that lays out actual requirements, getting out ahead of what TBS will hand 
down in 2019? 


e Cue up a senior management retreat on these issues? 


+ Something else? 


To what extent are people thinking about these issues in their day to day work, as they contemplate policy 
changes and look to redesign our program lines? 
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Notes to support PCO Consultation on Government of Canada Date Strategy March 22, 2018 


e iRCC is developing or experimenting with data-driven approaches in the fields of administrative decision-making, fraud 
detection and client service, among others. 


e Once ad hoc, IRCC is starting to view its innovation efforts through a digital transformation lens, For IRCC, this means: 


> Shifting our Department from a legacy approach to new ways of working and thinking using digital, social, mobile 
and emerging technologies 
Integrating digital technology into all areas of our business, resulting in fundamental changes to how we operate 
and how we deliver value to clients 
> Cultural change that requires us to continually challenge the status quo, experiment often, and accept that some 
ideas will fail 
e Advanced technologies, including advanced analytics, Al and machine learning, are seen as vital to the future success 
of the Department. We recognized that data is one of our most valuable assets, but also that we need to invest in the 
breadth, depth and reliability of our data sets. New tools are only as good as the data they draw upon. Data quality is 
probably the single greatest determinant of success in projects involving data analytics and/or machine learning. 


e Some major issues that IRCC would like to see a GoC Data Strategy address: 


> Data literacy for all public servants — not just the data science community, but also policy analysts, program 
advisors, senior managers, etc. 
> Infrastructure limitations, especially bandwidth and cloud storage — including, but not limited to, the role and 
capabilities of Shared Services Canada 
> Privacy/shareability across program lines and across departments (implementing, wherever possible, a “tell me 
once” approach). 
a Inan ideal world, IRCC would be able to access information such as entry/exit records and automated 
security assessments from the Canada Border Services Agency, or information about tax returns from the 
Canada Revenue Agency. A national Data Strategy could be a lever for legislative changes and/or 
information-sharing arrangements that may be needed to access data from other departments, or for new 
IT infrastructure to enable data transfer and processing. 


Question: fs this GoC Data Strategy exclusively inward-looking, or does it look outward as well? ie. Does it relate only to the 
activities of government, or also to how government regulates Canadian society and the economy to prevent harm to citizens? 


+ To the extent that the Strategy is outward-looking, we expect it would address Canada’s legal and regulatory frameworks 
for intellectual property, privacy, competition, consumer protection and human rights ~ the idea being that our 
frameworks are out of step with a world in which data is a resource and related policy challenges cut across legal and 
jurisdictional silos. 


e This component of a national Data Strategy would be concerned with policy innovation in the public interest, to avoid the 
barriers and uncertainties that come from a wait-and-see approach where a handful of multinational tech companies 
increasingly determine Canada’s data ecosystem. It would, ideally, focus on how the Government of Canada can ensure 
that citizens control the data they and their institutions create. 


e |IRCC would have a role to play in this part of a national Strategy because we are an identify management organization. The 
Department holds one of the most extensive databases in the Government of Canada: tens of millions of passports and 
” other travel documents issues to Canadians, and records on even more foreign nationals, many of whom will become 
permanent residents and citizens. All of this data must be stored securely, managed to safeguard privacy, and leveraged 
responsibly to deliver a client experience comparable to what people have come to expect from the private sector. IRCC 
can’t do any of this alone; our success depends on partnerships with CBSA, ESDC, foreign governments, provinces and 
territories, airlines, educational institutions and others. 
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“Update: Advancing a Policy and Legal 
Framework to Support IRCC’s Digital Transformation” 


Background Information to Support Presentation to ExCom 


In addition to the speaking notes embedded in the deck, the following has been developed to 
support your presentation to ExCom on the policy and legal framework for electronic tools. 


Slide 3 — “Context” 


e When this deck was presented to the DG Steering Committee and at Policy Committee, 
participants flagged that IRCC’s digital transformation will have significant impacts 
beyond the operational, legal and policy spheres—such as on human resources, IT 
infrastructure, and program integrity. 


o While this is true, it should be emphasized that SPP Branch’s work is limited to 
the policy and legal framework, and that these other issues will need to be 
considered and addressed by other areas of the Department. 


‘Slide 4 — “Review of IRCC’s Legislative and Policy Framework (Summer/Fall 2017)” 


e Part 4.1 came into force on October 18th, 2017 and provides explicit authority for IRCC 
to make use of automation and electronic tools in immigration lines of business, 
including automated decision-making. 


N/R t 


e Departmental consultations involved circulating our issue papers across IRCC’s policy 
and operations sectors, Legal Services, SIMB, and Corporate Affairs Branch. Through this 
process we received working-level feedback which we incorporated into the papers. 


e These papers were then rolled up into a deck which we shared with the DG steering 
committee and Policy Committee; feedback from these discussions have been 
incorporated into the deck for ExCom. 
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èe In terms of external consultation, SPP has been engaging with Michael Karlin at TBS who 
is leading the development of a white paper on the responsible use of artificial 
intelligence and automation in government. The issues he mentions and the solutions 
he proposes in this white paper align very well with our work. SPP has also participated 
in an interdepartmental dialogue on artificial intelligence/automation led by Policy 
Horizons and includes representatives from the Department of Justice, Innovation, 
Science and Economic Development, and others. Discussion at this meeting confirmed 
that we are looking at key issues that need to be addressed as we move to machine- 
assisted processing and decision-making. 


NR Slide 5— 


e Part 4.1 provides broad legal coverage, including for OPP’s work on the China TRV pilot 
by providing authorities for the use of electronic systems to process applications; 
however, it was put in place based on what was envisioned in 2014-2015. 


N/R 
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Slide 6 — “Issues: Fairness and Transparency” 


N/R 


Insights from this analysis can be built upon as similar analysis are 
undertaken for new uses of technology at IRCC. 


While LSU is responsible for advising the Department (especially Operations Sector) on 
legal and litigation issues related to specific technology-based projects, SPP will play a 
key role in bringing the key players together, building a knowledge base/centre of 
expertise and promoting consistency in the Department’s approach to these legal and 
policy issues as they are addressed in the context of each new project. 
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Slide 8 — “Issues: Governance (Internal and External)” 


SPP Branch’s Digital Policy team has been working closely with OPP and Legal Services 
Unit (LSU) to identify and address legal and policy issues related to the China TRV pilot. 
This cross-sectoral engagement should be considered a model for collaboration with 
internal partners on new uses of technology. 
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HIGH-LEVEL SUMMARY OF DISRUPTIVE TECHNOLOGY INITIATIVES AT IRCC January 2018 


STE 
s.16(2) 


1. ADVANCED ANALYTICS FOR IMMIGRATION APPLICATIONS 


Starting in March 2018, online visa applications from China will be triaged by a model so that low-risk cases can 


Based on recent pilots, it is expected that the use of advanced analytics will boost processing capacity and 
generate efficiencies. Deploying predictive models will allow officers to focus their attention on more complex 
cases, thereby strengthening program integrity, and faster application processing will shorten wait times and 
contribute to an improved client experience. 


2. AN AL TOOL FOR IMMIGRATION LAW 


IRCC and the Department of Justice are partnering in the development of artificial 
intelligence (Al}-powered tools to assist in immigration law. Such tools are expected to 
generate new insights and efficiencies in litigation management and the delivery of legal 
services. Specifically, the departments are seeking to develop, for discreet immigration law 
questions, tools that will: 


4. Support case law and legal research 


2. Predict litigation outcomes and assist in the development of legal advice/legal risk assessments 


3. Facilitate trend analysis in litigation to inform litigation strategies; for example, insights can be gained into 
key factors influencing the court in litigation that human review would not have revealed 


Department of Justice counsel and paralegals support IRCC officials in reviewing, responding to and managing 
very high volumes of immigration legal issues and litigation. Some work that previously would have taken these 
officials hours to perform could be performed in minutes. In addition, the analysis performed would be more 
detailed and comprehensive due to the added power of machine learning technology applied to large data sets 
of information (e.g. reviewing all key facts of every case decided on a given legal issue). 


Al tools for other areas of law have been developed by private industry but not for immigration law in Canada. A 
Request for Information {RFI} will be posted in early 2018, seeking industry feedback and insight into the 
development, use and scalability of Al-powered tools for immigration law. Employment and Social Development 
Canada will be joining this RFI to seek feedback from industry on Al tools for employment law. 


The Department of Justice has set up a Task Force on Al to survey the current state of Al in the legal profession, 
its existing and potential applications, and the challenges and opportunities it may present. 
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Think piece for senior management discussion 


TRANSFORMATION AT IRCC: STRATEGIC FRAMING 


CONTEXT 


Following years of aggressive growth in temporary resident lines of business, and with simultaneous expansion on 
the permanent resident side, IRCC’s processing network is stretched to the limit. Addressing these pressures with 
traditional reinforcements — more officers and more physical space ~ is not sustainable in the long term. The 
Department is adapting: we now have a Cabinet-approved Multi-Year Levels Plan, which better equips IRCC to 
manage inventories and meet permanent resident targets, and we have an integrated network vision to distribute 
workload across the Centralized, Domestic and International Networks according to each group’s intake capacity 
and place-based expertise. These advancements notwithstanding, meaningful innovation in business processes is 
required to keep processing times and backlogs from increasing across all business lines. 


Compounding the volumes issue, client expectations for quick and user-friendly services are increasing, in no small 
part because of the ubiquity of technology — in particular the rise of cloud, social media and mobile technologies 
that are changing the way we do things in every aspect of our lives. Worldwide demand for air travel continues to 
escalate, and the rise of a global middle class is swelling the ranks of mobile, tech-savvy individuals who are 
accustomed to a certain standard of digital interaction with private sector entities and now expect the same from 
governments. 


Canada’s international competitors are increasingly leveraging technology to streamline business practices in order 
to better attract tourists, business visitors and skilled permanent residents. As safety and security threats change in 
scope and complexity, like-minded countries are using new tools to analyze and manage risks. For Canada to keep 
pace, harnessing the potential of emergent technologies and other innovations will be critical. 


The bona fide arrival of powerful technological capabilities brings with it significant opportunities for IRCC. Once 
largely theoretical, or at least the sole domain of leading-edge tech companies, the dramatic pace of technological 
change has brought tools such as digitization, data analytics, automation and machine learning into the 
mainstream, Recent years have seen a dramatic decrease in the cost of prediction due to advances in 
computational speed, data storage, data retrieval and algorithms. More widespread understanding of the 
technology is leading to commonplace use of “big data” — in health care, in education, in government 
administration of all types. 


With this technological advancement has come a broader and more generalized focus on innovation, including 
within the Government of Canada, where it is now seen as a priority for most federal organizations. With 
leadership from the Privy Council Office’s Innovation and Impact Unit and centres of expertise in various federal 
departments, including IRCC, organizations are leveraging new approaches, such as data analytics and user-centred 
design, to revolutionize service delivery models. 


WHERE WE ARE GOING 


IRCC has begun to take critical steps to experiment with new technologies and other innovative approaches to 
respond to external pressures and deliver on its mandate. The Department has created pockets of expertise in 
novel areas such as advanced analytics and behavioural insights, which, respectively, have the potential to play a 
significant role in reducing processing times and enhancing the client experience. Work is also underway to update 
and “future-proof” program design in key areas such a visitor visas, identity management and security screening. 
More broadly, our processing network is being examined across business lines to develop a more integrated and 
efficient service delivery model. This is on top of years of determined work to better manage volumes through 
better intake controls, inventory blitzing, and levels planning management. 
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The Department is now at a point in its maturity to fundamentally change the way it does business. Several key 
initiatives already underway are potential catalysts for a larger transformative shift: 


The Advanced Analytics Project: The China TRV Pilot and Beyond 


The Department reached a major milestone in 2017 with a highly successful pilot of a state-of-the-art predictive 
model for risk triaging and automated approval of low-risk online applications (e-Apps). IRCC piloted a predictive 
model to sort China visa applications into tiers — the lowest risk for auto-approval, medium and high risk for officer 
review. Over a three-week pilot in October/November 2017, the model took 42% of all e-Apps off of officers’ 
hands, demonstrating the tremendous potential of this technology for relieving pressure on the processing 
network once e-Apps become the dominant application mode. 


in March 2018, the model will be deployed in China as a permanent tool for daily processing, ` 


‘he 
Operational Planning and Performance Branch will establish and grow an Advanced Analytics Lab to test more new 
ideas and concepts. in time, with advanced analytics used across the majority of business lines, the Department 
will find its processing model transformed. 


A Focus on Client Experience 


in January 2017, Client Experience Branch (CEB) was established to oversee the Department’s Service Excellence in 
Action agenda, which focuses on the four principles of innovation, risk management, client experience and public 
confidence. With the understanding that service excellence is a shared responsibility across the Department, CEB 
brings leadership and expertise to develop client insights, drive user-centred experimentation and innovation, 
report on client impacts of ongoing initiatives and provide horizontal guidance to the Department, including 
through a departmental Service Strategy. CEB is also accountable for IRCC’s primary front-end service model, 
including the Client Support Centre, which is expected to provide high quality and consistent client service by 
phone and email. With advanced analytics bringing transformational change to IRCC’s back-end process, the work 
of CEB can have a similar impact on forward-facing activities. 


N/R 


A Horizontal Policy and Legal Framework Underpinning Automation, Analytics and e-Tools 


Strategic Policy and Planning Branch, working in concert with Legal Services, is working to identify and address the 
policy and legal issues related to uses of technology. With the benefit of both enabling and challenge functions, the 
Department will be better able to determine what uses of big data, machine learning and automation are possible, 
permitted, and advisable. Established links with centres of expertise in central agencies (notably TBS) and the 
research community (notably Policy Horizons) will serve to bolster IRCC’s policy capacity in this emerging area. A 
new Director General Committee focused on IRCC’s e-Tools policy and legal framework can provide leadership as 
policy specialists begin to generate horizontal advice to inform the design of technology-related initiatives. A White 
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Paper in early 2018 will help to cement IRCC’s status as a Government of Canada leader with respect to innovation 
and the responsible use of emergent technology. 


While these initiatives are important pieces of the puzzle, a full-scale transformation will require work in other 
areas as well. Exploratory work is ramping up as follows: 


A Future Vision for Visitors 
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À cohesive transformative vision will depend on understanding and systernatically managing the linkages 
between these various initiatives. 


disadvantaged if our governance structures are not in step with the changing and increasingly significant role of 
IT in policy, program and operational changes. 


The Department is at a critical pivot point. If we can find a way to integrate all of the initiatives described in this 
paper into a coherent implementation plan, we can begin to effectively tackle volumes, offer a client experience 
that lives up to applicants’ expectations, and deliver on our mandate to provide a safe and secure immigration 
program for Canadians. 


DISCUSSION QUESTIONS 
> Do we buy into this description of IRCC’s current state and vision for where we are going? 
> lf forced to choose, do some of these initiatives take priority over others? 
> Are the leads clear for each strand of work, or do we need to build more “connective tissue” (e.g. 
working groups) around these? How can IRCC formalize linkages to ensure we maximize potential 
benefits? 
> Does the Department have the necessary expertise to achieve full-spectrum change? What functions 


could be or should be outsourced to third parties? 


» What engagement is required with other government departments (particularly CBSA given the critical 
importance of alignment), partners and stakeholders, and the public to support a successful 
transformation? 
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Al Interdepartmental Dialogue Meeting — Oct 3, 2017 8.23 


While we’re not yet currently in the world of Artificial Intelligence at IRCC, we could be moving in that 
direction and we feel that some of the activities currently underway raise many of the same policy and 
legal questions as the use of Al would. 


So what is IRCC doing? We’re using of predictive analytics to help us processing immigration applications 
more quickly. Specifically, a group of data scientists in the Department has developed a predictive model 
to triage temporary resident visa applications based on risk. This model is currently being tested in China 
where it will identify low-risk cases that can be bulk approved without an officer review. In other words, 
large numbers of low-risk applications can be approved en masse without having to be reviewed by a 
human being. 


The model functions by mimicking officer decisions about whether an application should be approved or 
not and tries to predict the final decision that an officer would have made. To do this, the model analyzes 
thousands of past officer decisions to understand how combinations of applicant characteristics are linked 
with approval or refusals. Patterns which yielded almost unanimous agreement among officers in favour of 
approval are used to recommend future approvals (“green bin”). The rest are triaged to officers for review. 


This approach was initially tested in China in December 2016 and is going to be tested on a somewhat 
larger scale this Fall. Based on results of the initial testing, it is expected that the results will be positive 
and that there will be pressure to begin rolling out this approach more broadly given that demand for 
temporary resident visas is forever on the rise. 


In 2015, changes were passed to the Immigration and Refugee Protection Act that gave IRCC broad 
authority to use electronic tools, analytics and automation to processing immigration applications. These 
changes are not in force yet but will be published in the Canada Gazette on Oct 18". : 


These are some of the questions we’ve been considering, among others, and we do a lot of discussing 
them among ourselves but we need outside perspectives, so we jumped at the opportunity to meet with 
this group. So with that, if anyone has any questions or guidance or anything else, we’d love to hear it. 
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s My DGO has reached out to many of your DGOs to ask that you provide a point of contact for this consultation. 


Notes on the White Paper 


e Last summer, your branches were consulted on the development of a series of issue papers related to new uses of 
technology. At the same time, TBS/Michael Karlin was developing a white paper on responsible use of artificial 
intelligence in the Government of Canada. 

+ SPP Branch has built on this earlier work, as well as lessons learned from OPP’s successful China TRV pilot, to 
develop our own white paper that frames the key benefits and risks of IRCC using emergent technologies. 
Primarily, we're talking about big data, automation and artificial intelligence, and principles and considerations for 
IRCC as it makes increasing use of these technologies. 

* SPP has been consulting areas of the Department whose work is specifically referenced in the paper, and then 
we’ll be reaching out to TBS and other departments who are doing work in this space for an external perspective. 

e We will be looking for a way to officially “launch” the white paper, such as taking it to Policy Committee. 


NEXT STEPS AND FORWARD AGENDA 


ə This Committee will meet again in the spring. 


5 mins 


+ Agenda items may be brought to Amanda McPherson, Assistant Director, Digital Policy, SPP Branch 
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Notes on the TBS Automation Standard: 


e TBS recently hosted an interdepartmental consultation to launch the development of a standard for departments 
pursuing automated decision-making. 

e Scope: anything from process automation to Al; focus on administrative decisions that flow from legislation; 
broader than automated decision-making ~ includes decision support (e.g. system-generated recommendations 
to humans) 

e TBS would like to ensure that Departments are sufficiently transparent in the use automated systems, and that 
they comply with ethical guidelines 

e SPP and OPP both attended the kick-off meeting and will continue to support this work 

e it's a two-way street: IRCC wants early warning of rules that might be coming, but we also have expertise to share 
and an interest in influencing this government-wide effort 

e TBS aims to share v1.0 of its proposal with Chief Technology Officers this winter, then final approval by TB 
Secretary in the spring 
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TERMS OF REFERENCE 
DG Steering Committee: Digital Transformation Policy & Legal Framework 
Documents: Terms of Reference 
Notes: 
+ The Committee provides a cross-sectoral focal point to discuss policy and legal issues related to IRCC’s use of new 
technology. 
e = |tis atime-limited undertaking — not part of IRCC’s ongoing governance structure. 
e Not all new uses of technology are in scope — focus is on technology that transforms program design, necessitates 
new policy or legal authorities, or that could tead to legal challenges. Matt de Vlieger, SPP 15 mins 
> For example, the automation of routine clerical tasks would not be in scope {like auto-promotion of applications in f 
GCMS). This committee would be concerned with uses of technology that in some way influence, directly or 
indirectly, decisions on clients’ applications (for example, by automatically triaging applications into 
low/medium/high risk categories). 
e itis requested that project leads whose work falls within scope present their plans to the Committee in the early 
stages of development. 
Questions: 
+ Do you support the Committee’s mandate as stated in the draft Terms of Reference? 
œ Does the Committee membership include the necessary players? 
ROUND TABLE AND UPDATES ON KEY FILES 
Three things for SPP to mention: 
1. TBS Standard on Automated Decision Support Systems 
All 25 mins 


3. An {RCC White Paper — Digital transformation at IRCC: Benefits, risks and guidelines for the responsible use 
of emergent technologies 


OPP (attendance TBC) should be able to give a debrief on the February 5 Min-DM. 
Admissibility Branch will provide a debrief on the Visa Innovation DG Lock-Up. 
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February 9, Z018 3:00-4:000m 
JETS A1801 


ANNOTATED AGENDA 


Item 


WELCOME AND TOUR DE TABLE 


POLICY AND LEGAL FRAMEWORK FOR IRCC’S DIGITAL TRANSFORMATION — DEBRIEF FROM EXCOM 


For reference: Deck to ExCom 


Notes: 


Presentation was well-received by ExCom. 

Members agreed to formalize a multi-sectoral DG Committee. 

Recommended that IRCC engage external partners (in particular the CBSA) on changes to policy and legal 
framework to broaden our use of technologv. 


technology use could produce cost savings tor IKCU, with impacts on our tee structures. 
o Finance to be engaged on the DG Steering Committee as needed. 
Agreement that the Minister should be briefed on the China TRV Pilot and how IRCC is addressing related policy 
and legal issues. 
o OPP presented on Feb 5 and SPP included content in their deck on how policy and legal issues have been 
addressed. 
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Presenter Time 
Matt de Vlieger, SPP 5 mins 
Matt de Vlieger, SPP 10 mins 
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SCOPING A DIGITAL TRANSFORMATION WHITE PAPER Nov 2017 


Mandate 


Numerous initiatives around the Department, ranging from client service to identity management to stakeholder 
collaboration, involve new technology in some capacity. Judging whether a particular technology is in scope for 
the Digital Policy team, and for the white paper we intend to bring forward, can be challenging. In making these 
determinations, we propose the following four tests: 


1. Does the technology fall within the definitions of data analytics, automation or artificial intelligence? 


2. Could the technology have a disruptive or transformative impact on the way the Department does 
business? 

3. Do applications of the technology relate to outward-facing activities that are governed by the 
Department’s legal framework (i.e. activities that affect, if indirectly, external clientele — as opposed to 
strictly internal business such as HR and financial management}? 


4. Are applications of the technology proximate enough? Is piloting or implementation conceivable within 
the next three years? 


if the answer to each of these four questions is yes, the Digital Policy team has a role to play in determining 
whether there are ethical, legal and policy issues that have yet to be fully considered. 


Audience for a white paper 


Core audience: Secondary audience: 
> Project Leads > iT 
> Modellers > HR 
> Senior Management (abridged version) » Program integrity specialists 
> Policy Analysts 4 ATIP | 
> Legal Services > Security 


Not targeted at: Administrative decision-makers, officers in the field, programmers 


> Though the content of the white paper will eventually lead to and provide a basis for guidance to officers 


Type of advice 


1. Horizontal/functional advice focused on policy and legal issues related to uses of technology 


ns 


> NOT vertical/program policy advice on what direction to take IRCC programs ~ but can inform 
program policy i 
2. Enabling and challenge functions: 


identifying policy and legal risks, requirements, what is possible/perrnitted 


> identifying opportunities, best practices, what is possible/promising, what is advisable 


3. Identifying linkages to other functional areas (e.g. HR/change management, IT, program integrity) 


> NOT assessing or advising on specific impacts or possible approaches in these areas 


A-2018-09320-001385 


| $ | Immigration, Refugees Immigration, Réfugiés 
and Citizenship Canada et Citoyenneté Canada 

Information disclosed under the Access to Information Act 

L'information divulguée en vertu de la loi sur l'accès à l'information 


DG Steering Committee: Digital Transformation Policy & Legal Framework 
Terms of Reference 


Mandate 


The Digital Transformation Policy and Legal Framework DG Committee (the Committee) provides a 
cross-sectoral focal point to discuss policy and legal issues related to IRCC’s use of new technology. The 
Committee supports the Department regarding what uses of big data, advanced analytics, automation, 
and artificial intelligence are possible, permitted, and advisable for IRCC. It is a temporary body intended 
to support the Department during this time of transformation, and is not part of the Department’s 
ongoing governance structure. 


Scope 


The Committee’s focus is on technologies that have the potential to transform the way the Department 
does business; for example, technology that transforms program design, necessitates new policy or legal 
authorities, or that could lead to legal challenges. Technology that involves the simple replication of 
current processes using electronic systems or tools is out of scope. The Committee will manage any 
overlaps in mandate or other relevant linkages with other internal DG-level committees, such as the 
information Sharing and Privacy Committee and the DG Information Management/Information 
Technology Committee. 


Responsibilities 
All Committee Members 


e Ensure policy and legal issues related to new uses of technology are identified and addressed at 
the design and development stages of new projects; 

* Provide strategic leadership and a necessary challenge function on the use of new technology; 

e Provide a focal point for continued cross-sectoral engagement, information-sharing, knowledge 
building, and sharing of lessons learned around legal and policy issues related to new uses of 
technology. 


Project Leads 


* Present plans related to new uses of technology in the early stages of pilot/project development 
to ensure legal and policy issues are flagged ahead of approval, testing, and/or implementation. 


Committee Membership 


Chair 

The Committee is chaired by the Director General of Strategic Policy and Planning (SPP) as the branch is 
leading efforts to put in place a policy and legal framework to support the broader use of new 
technology across program lines and IRCC’s overall digital transformation. 


Last updated: February 9, 2018 
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Regular Members 
Membership includes Directors General from the Policy, Operations and Corporate Sectors, as well as 
senior representation from Legal Services. 


Policy Sector 


Director General, Strategic Policy 

Director General, Admissibility 

Director General, Citizenship 

Director General, international and Intergovernmental Relations 
Director General, Immigration Branch 


Operations Sector 


Director General, Operations Planning and Performance Branch 
Director General, Immigration Program Guidance 

Director General, International Network 

Director General, Centralized Network 

Director General, Domestic Network 

Director General, Citizenship and Passport Program Guidance 
Director General, Case Management Branch 

Director General, Client Experience Branch 

Director General, Migration Health Branch, 

Director General, Operations Strategic Projects Office 


Corporate Services Sector 


Other 


Director General, Corporate Affairs 
Director General, Solutions and Information Management 


Counsel, Legal Services 


*invitations will be extended to other DGs based on subject matter 


Secretariat 


SPP will coordinate the planning and preparation for meetings of the Committee. 


Designated Alternates, Observers and Presenters 


Membership will be at the Director General (DG) level. 


Observers are permitted for specific agenda items as required. 
Other representatives may be invited to present or participate as required. 


Last updated: February 9, 2018 
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BRANCH: Strategic Policy & Planning DATE: March 14, 2018 


À WHITE PAPER ON DIGITAL TRANSFORMATION AT IRCC 


ISSUE 


e To provide you with a draft of SPP Branch’s think piece entitled “Digital Transformation at 
IRCC: Benefits, risks and guidelines for the responsible use of emerging technologies” 


BACKGROUND: 


e Ata December 4, 2017 meeting of ExCom, SPP Branch provided an update on its work to 
ensure that the Department’s policy and legal authorities support broader use of technology 
across all program lines. 


e As part of its forward plan, SPP Branch committed to developing a white paper in 2018 to help 
the Department think through the wide-ranging implications of technological innovation. 


e Treasury Board Secretariat has developed and widely distributed a White Paper on 
Responsible Artificial Intelligence in the Government of Canada. Building on this piece, and on 
a series of issue papers developed by SPP Branch’s Digital Policy team (with input from 
across the Department) in the summer of 2017, this paper focuses on IRCC’s specific set of 
challenges and opportunities. The paper includes a discussion of artificial intelligence, but 
casts a wider net, incorporating also data analytics, automation and other electronic tools. 


e The paper aims to: 
~ Boost the Department's digital literacy; 


— Give people a common lexicon and framework for understanding the specific technologies 
that are relevant to IRCC’s operations, as well as a basis for weighing the risks and benefits 
associated with each; 


— Provide interim guidance to program designers, policy analysts and senior managers; and 


~ Kickstart a larger effort to ensure that as the Department continues to use technology in 
new ways, we ensure fairness for applicants, we mitigate new and unexpected risks, and 
we realize the full potential of transformative tools. 


CURRENT STATUS: 


e This is a first draft of the paper. While SPP Branch has consulted with a small group of subject 
matter experts (some internal and some in other federal government departments), there are 
still a few place-holders where input from colleagues is outstanding. 


e Contacts in the Operations Sector have expressed an interest in giving the paper to ADM 
Kochhar as soon as possible, in light of that sector’s ongoing work to develop a Five-Year 
Strategic Plan. This is why we are providing a draft to you now, rather than waiting to send a 
final version. 


NEXT STEPS: 


e SPP/Digital Policy will continue to refine the paper, with a view to launching it Department- 
wide in the coming weeks via Policy Committee and other channels, such as the recently 
formalized DG Committee for a Digital Transformation Policy and Legal Framework. 
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BACKGROUNDER ON AUTOMATION Strategic Policy and Planning Branch 
January 2018 


WHAT IS AUTOMATION? 


Automation is the training of computers to perform routine and repetitive tasks under conditions and ina 
manner pre-determined by people. 


> At present, the Department is mainly interested in automating non-discretionary or easily defined steps in 
processing applications and communicating with clients, thereby freeing up officer time for more value-added 
activities, such as investigating complex cases or conducting quality assurance exercises. 


Automation is often used as a catch-all term, encompassing a whole continuum of activities, from computerizing 
clerical tasks to automated decisions based on predictive analytics. 


> Automation of routine, clerical tasks, while important, raises fewer issues from a legal standpoint. When a 
computer promotes applications to GCMS, this is an example of routine, clerical automation. 


> We begin to cross the threshold from routine automation into automated decision-making when, for example, 
we use complex business rules to allow a computer to automatically triage applications based on their degree 
of risk. 


> When the computer isolates applications tagged as “very low risk” and auto-approves them without an officer 
exercising a judgment function, we are fully in the realm of automated decisions. 


> As we move along the automation continuum from the routine to the complex, we begin to see greater 
potential consequences for clients (and, by extension, for legal and policy frameworks) and for IRCC business 
processes. 


PROGRAM LEADS SHOULD FLAG AUTOMATION PROJECTS THAT COULD HAVE DIFFERENTIAL IMPACTS 
ON CLIENTS 


Some automation proposals can seem innocuous at first glance, but turn out to have significant legal and policy 
implications. In identifying automation projects in this category, program leads should ask themselves, “Does the 
proposed machine intervention have any role to play in determining the outcome of a client’s case? Could these 
changes potentially influence whether applications are ultimately approved or refused?” if the answer is yes, 
programs leads should consult the Digital Policy group in SPP Branch. A legal opinion may also be required ensure 
that program-specific implications are identified and addressed. 


in preparing for greater use of automation and automated decision-making, the Department needs to 
think carefully about how to: 


— ensure procedural fairness for applicants 

— explain outcomes to clients and demonstrate accountability for decisions 
— create an audit trail in the event of a legal challenge 

— prevent technological supports from fettering officer decision-making 

— ensure that information retention and sharing respect privacy legislation 


Note: SPP Branch is developing a white paper on the responsible use of emerging technologies, including data analytics, 
automation and machine learning. The paper will provide a detailed treatment of the pertinent policy and legal issues in early 
2018. In addition, consideration is being given to formalizing a DG committee to guide the development of a legal and policy 
framework to support digital transformation. 
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Electronic Tools, Automation, and Advanced 
Analytics at IRCC 
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SECRET 


To provide an overview of: 


> three “game-changing” technologies that could help Department improve 
how it does business, 


current examples and potential future uses of these technologies at IRCC, and 


key considerations related to greater use of technologyin delivering IRCC’s 
programs and services. 


With a number of Departmental priorities having a technology angle — e.g. traveller 
facilitation and client service — it is an opportune time to explore whether IRCC as a 
whole should pursue greater use of technology and data to in the medium-term to 
improve how it does business more broadly (i.e. not simply in a piecemeal fashion) 
This deck provides an overview of what this could look like, what the benefits could 
be, and what we need to do as a Department to advance our use of technology and 
data, particularly with respect to ensuring legal authorities are in place to support 
this direction. 

The Government’s focus on innovation and experimentation could be an 
opportunity for IRCC to move forward on a lot of the thinking and work it has done 
to support technological advancement. 

SPP has consulted with OPP, Admissibility and [SIMB?] in developing this product, 
but we need broader input as well and that’s what we’re hoping to achieve today. 
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IRCC is reducing processingtimes via expanded admissions space, inventory 
reduction, and renewed investments in its current processing system {including 
funding secured in the 2016 Levels Pian). 


But, keeping pace with demand will not be sufficient to ensure the success of our 
immigration system: 


* Clients expect fast, streamlined, electronic service, and the Government has made ita 
priority to improve client service and expand digital platforms to meet these expectations; 


Canada's international competitors are Increasingly using technology and new approaches 
to improve service delivery to attract tourists, business visitors, and global talent; 


Safety and security threats are growing in significance and complexity, necessitating the use 
of new tools to ensure the safety and security of Canadians and maintain public confidence. 


The Department’s chief assets ~ human resources, data and 
information technology (IT) — are in a constant state of evolution and 
need to be kept relevant in this changing global landscape. 


- Over the past number of years, and also more recently over the course of 2015 and 
2016, the Department has significantly advanced its ability to process an ever- 
increasing number of applications in a timely fashion. 

- However, as demand continues to grow (and we expect that it will — if not for PR, 
certainly for TR), the way IRCC does business will need to evolve in step with its 
evolving operational context: technology is everywhere and, as a result, clients 
expect fast, electronic service; international competition will persist, with our 
competitors using the benefits of technology to help attract visitors and global 
talents; and safety and security threats are changing in scope and complexity and 
technology will be critical in our response to this shift. 
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- Data & Predictive Analytics: Using computer systems to analyze 
large quantities of data to identify trends and risk indicators in order 
to improve decisions how we make decisions on applications. 

- Automation: Using an electronic system to perform repetitive, non- 
discretionary tasks, like completing steps in application processing. 

- To varying degrees, we already use these tools at IRCC as we’ll see on the next few 
slides, as well as additional ways that they could potentially be deployed. 
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Current Uses 
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Current Uses 
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Taken together, these changes would lead to an increasingly virtual program delivery 
model where IRCC and its officers harness structured data and information-sharing to 
manage volumes and risk while facilitating the majority of applications so that: 


Most clients... Officers 
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Using more technology and becoming a data-drive department will result in changes 

for clients, officers, stakeholders and our delivery partners. 

e Clients will receive better, faster, more individualized online services, while also 
having their data used in more and different ways to improve the services we 
provide to them but also our own internal processes 

* Officers will be supported by data and spend less time on routine processing steps, 
focusing more on value-added activities 

+ Stakeholders — like employers and universities, and sponsors of family members and 
refugees — will benefit from faster processes and, in some cases engage in two-way 
information-sharing with the Departments, such as universities sharing registration 
information with us and IRCC confirming status with them via an electronic portal 

* Delivery partners will have to adapt and keep up with IRCC’s technological 
advancement, but will also benefit from it through greater access to and more 
streamlined provision of data. 
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In 2015, a series of new provisions 
under Part 4.1 of the Immigration and 
Refugee Protection Act (IRPA) set out 
clear authorities for the use of 
electronic tools and systems 
{including automation), and gave IRCC 
authority to make regulations 


governing the use of these systems, 


These amendments are not in force; 
when they are, they will allow IRCC to 
further leverage technology to 
improve processing efficiency, client 
service, and program integrity in the 
temporary and permanent resident 
business lines. 


- In 2015, IRPA was amended to include a series of provisions allowing the IRCC to use 
electronic tools and systems in its operations - basically, these provisions allow the 
Department to carry out any of its TR and PR operations using an electronic tool or system, 
clarifying the scope of its authority with respect to the use of technology. 

- However, these amendments are not yet in force - before pursuing technological 
advancement for client service transformation or other improvements, we need to bring 
them into force. 

- [Memo status?] 


More details on amendments if required: 


The amendments to /RPA: 
"Clarify authorities to use electronic application, processing and payment systems in 
all TR and PR lines of business and explicitly permit the use of automated-decision 
making; 
"Establish the authority of the Minister to make the use of an electronic tool or 
system mandatory through regulations; 
=Provide a new authority for the Minister to direct the use of any means of application 
in circumstances defined by regulation. 

The amendments to /RPA include regulation-making authorities to: 
=Specify details regarding the technology and systems to be used and matters relating 
to that system (i.e. the date/time when electronic documents or information are 
deemed to be sent or received); 
"Require the use of an electronic system or means (including an e-payment system), 
as well as to specify exceptions to that requirement (e.g. in cases of disability); 
=Prescribe circumstances where the Minister may direct the use of alternate means 
(e.g. in cases of natural disasters); 
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To date, IRCC ‘s IT development has focused on functionality and efficiency of 
back-end systems designed to collect, process and store applications. 


User (officers or applicant) experience has not been a primary focus given: 


historical mandate concerned more with risk mitigation than facilitation; 

emphasis on financial efficiency rather than the creation of value-proposition; 
project approach that has allowed little ongoing user testing & adjustments; and 
massive change agenda as IRCC adopts e-processing across all of its operations while 
safeguarding client data and maintaining system stability. 


As global competition for skills increases and client expectations evalve, the 
Department is increasingly exploring e-tools that mirror private-sector 
counterparts to deliver an effective and welcoming user interface. 


IRCC will need to determine if 3rd party delivery models will be used to achieve e- 
service objectives in the way that Visa Application Centres (VACs} have been used 
to complement in-person service objectives, 


IRCC’s recent Family Class and Citizenship Grant design challenges show how 
human-centered design can harness low-cost technology to improve user 
experience and process efficiency, 


To date, IRCC’s service delivery has been largely focused on processing efficiency 
(“back office”) rather than client experience, given an emphasis on risk management, 
financial constraint, and a significant change agenda across the Department’s 
operations. 
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As information technology (IT) becomes the dominant method of IRCC program 
delivery, implementation of new initiatives will increasingly depend on whole-of- 
Department engagement in technology-related decisions. 


2 Policy and program development must take into account the limits of IT capacity, feasibility, 
and functionality ~if cannot be an afterthought. 


Many key policy and legal questions will arise and need to be addressed during the IT 
development process ~ program and policy areas will require iT literacy to effectively consider 
interactions between potential changes and enabling technologies. 


However, IRCC’s current governance structures do not take into account the 
changing and increasingly significant role of IT in policy, program and operational 
changes; the Department’s existing skill “siloes” are not sustainable as IT systems 
become central to service delivery, as well as program and policy design. 


Coherent technological change will require a governance mechanism to not only 
align Departmental priorities and policy initiatives, but also to integrate policy, 
legal, operational, and IT expertise. 
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IRCC will need to explore a number of key questions to ensure that the use of 
technology helps rather than hinders its ability to deliver on its mandate. 


How W Censure procedural fairness 
and mitigate other legal and litigation risks 
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* To update you on SPP Branch’s work to ensure IRCC’s policy and legal 
authorities support broader use of technology across all program lines, 
including a proposed way forward to ensure the Department's policy and 


legal framework evolves to support a transformation of its program delivery 
model. 
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Worldwide, organizations are using technology to transform service delivery by 
providing online services, automating processes to speed service delivery, and using 
advanced analytics to improve service qualityand risk management. 


in response to steeply increasing application volumes, changing client expectations, 
and government commitments to faster and better service, IRCC has been making 
increasing use of electronically-enabled processing and decision-making to maintain 
service standards. 


This transformation is having far-reaching implications, including for IRCC’s iT 
infrastructure, program integrity, workforce and partners, as well as for the 
Department’s policy and legal framework. 


SPP Branch is leading efforts to identify and address the policy and legal 
implications of this transformation to ensure IRCC has the right policy and legal 
authorities, support and guidance as we continue to use technology in new ways. 
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FIRCC Legislative & Policy Framework (Summer/Fall 2017) 


In April 2017, DM Morgan approved an approach to put in place a policy and legal 
framework to support greater technology use at IRCC. 


a Part 4.1 of IRPA was brought into force in October 2017 providing broad Ministerial 
authorities for the use of technology, N/R 


* Jn summer/fall 2017 SPP Branch undertook a review of IRCC’s legislative and policy 
framework and considered broader questions related to IRCC’s plans by: 


© Reviewing literature and international best practices related to technology in delivering 
government services; 


Developing a set of issue papers to identify key policy and legal issues related to electronic 
tools, automation and advanced analytics; 


Consulting across IRCC on the issue papers to ensure comprehensive issue identification, 
including the establishment of a multi-sectoral ad hoc DG Steering Committee; and 


Engaging with TBS and the Artificial Intelligence interdepartmental Dialogue led by Policy 
Horizons, 


“Asa result of this process, key issues have been identified in the 4 following areas: 


‘Fores à 


eben 
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+ Current approaches to ensuring fairness and 
transparency revolve around officer-based 
decision-making. 
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issues: Governance {internal and External) 


Across IRCC, there is acknowledgement that policy 
and legal issues (like operational orif issues) are 
arising and need to be addressed as the service 
delivery model transforms. 

To-date, including these perspectives has been ad hoc 


and based on the best guess/risk assessment of 
project leads. 


Policy and legal experts need to be brought in at the 
early stages of projects to ensure that issues are 
identified and addressed. 


To achieve success with new uses of technology and consequent changes to our 
policy and legal framework, consulting with external partners (especially CBSA) will 
be critical given our shared role in administering IRPA/IRPR and delivering programs. 
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Ongoing policy and legal guidance 
on projects 

Engagement with vertical program 
afeas that are incorporating 
technology inte their future vision 


White paper to guide responsible 
use of technology and inform 
transition advice 


Engagement with the Centre {TES} 
to align with guidance 

Review to identify and address 
related privacy issues 

Develop transition advice regarding 


scope of legel/poticy framework 
changes 


See Annex A for potential work plan 


p Policy/Legal engage in i 


-tools development via 
ad hoc or existing 
structures. 


Enhance governance 
through formalization of 
current multi-sectoral OG 
Steering Committee on E- 
Tools Policy/Legal 
Framework {see Annex 8 
for potential Terms of 
Reference} 


External partner 
engagement 
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Are these the right areas of work? Are there other policy and legal issues that need 
to be prioritized? 


Do you wish to engage the Minister on these policy and legalissues, or to focus that 
engagement via transition advice? 


Do vou concur with the formalization of the DG Steering Committee on E-Tools 
Policy/Legal Framework? 


NEXT STEPS: 


» With ExCom endorsement, continueto advance the policy and legal framework to 
support IRCC’s digital transformation, including: 


à carrying out work plan activities; and 


o improving governance around policy and legal components through the DG 
Steering Committee. 
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MANDATE: Continued engagement, information sharing and knowledge-building around legal and 
policy issues related to new uses of technology {including electranic tools, automation, advanced 
analytics}. 


SCHEDULE: Ad hoc 
RESPONSIBILITIES: 


* Project leads are expected to present plans related tc new uses of technology in the early 
stages of pilot/project development to the Steering Committee to ensure legal and policy issues 
are flagged ahead of approval/testing /implementation. 


Leads can share policy and legal issues, mitigation strategies and solutions to help support 
development af a shared, coherent policy and legal framework underpinning these projects. 


CORE MEMBERSHIP”: 
Policy Sector: DG-SPP {chair}, DG-Admissibility, DG-Citizenship, DG4IR 
Operations Sector: DG-OPPB OG-IPG, DG-IN, DG-CN, DG-CPPG 
Corporate Services Sector: DG-CAB, DG-SIMB 
Other: Legal Services Unit 


*tnvitationa will be extended to ether DGs based on subject matter 
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MEMORANDUM TO THE ASSISTANT DEPUTY MINISTER, OPERATIONS 


INNOVATION PROJECT FUNDED BY THE CANADIAN SAFETY AND SECURITY 
PROGRAM TO SUPPORT PREDICTIVE ANALYTICS 


FOR INFORMATION 


SUMMARY 


e The Operations Planning and Performance Branch’s (OPPB) Business Research, Intelligence 
and Analytics Directorate has won a second grant from the Canadian Safety and Security 
Program (CSSP) to support innovative research. 


e Under the new project, entitled Chain of Trust, OPPB will receive $237,000 in 2017-18 to use 
predictive analytics to develop risk indicators for the Electronic Travel Authorization Program 
(eTA). 


e The Chain of Trust project builds on the success of the currently funded FASTER PrivBio 
project, slated to end in March 2017. 


BACKGROUND: 


The CSSP is a managed through Defence Research and Development Canada’s Centre for 
Security Science, which regularly launches Requests for Proposals to fund research projects that 
enable advancements in Canadian public safety and security. 


In 2015-16, OPPB was awarded $60,000 from the CSSP as part of a $1.2-million project called 
FASTER-PrivBio, which involves the CBSA, a private company and two universities. Slated for 
completion in March 2017, the project has successfully developed a prototype smartphone app that 
provides an innovative mobile end-to-end eTA application and border management screening 
process by leveraging the capabilities of the ePassport, smartphone technology, biometrics and 
self-service kiosks. This prototype uses innovative encryption algorithms and Privacy by Design 
Principles to ensure the security of corporate and personal information. 


CURRENT STATUS: 


OPPB and its partners were successful in obtaining funding for a new $1.3-million innovation 
project for 2017-18, called Chain of Trust: Collaborative Risk Assessment for Air Traveller 
Programs. The project will involve OPPB, the CBSA, WorldReach Software, Entrust Datacard, 
Face4 Systems and the University of Ottawa. OPPB will receive $237,000 in 2017-18 under this 
project. 
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The project builds on the spirit of the Beyond the Border strategy and on the proven fact that 
travellers are willing to volunteer private information in exchange for ease and speed at travel 
check points. The project aims to develop a prototype which will spearhead advancements in 
security and traveller convenience by: 


1. Leveraging FASTER-PrivBio capabilities to collect additional data early in the travel 
continuum, including: 


ind the online submission of the E311 Declaration Form prior to arrival 
in Canada. 


2. Consolidating the data collected on the traveller throughout the travel continuum into a new 
transaction record, enabling the continuous and incremental verification of the traveller’s 
identity and assessment of level of risk. 


3. A privacy-respecting, collaborative predictive analytics risk assessment by IRCC and the 
CBSA, leveraging historical traveller information and trip-related data collected throughout the 
journey. IRCC and the CBSA will each develop their own predictive models to assess the risk 
level of each traveller. IRCC’s risk assessments during eTA adjudication will be shared with 
the CBSA and incorporated into their own risk assessments. Additionally, as each traveller 
advances through the travel continuum, new data about the traveller will allow the CBSA to 
update its risk score and inform decisions about which travellers should be sent to secondary 
inspection at the port of entry. 


This innovative collaboration during risk triage will allow low-risk travellers to benefit from 
faster processing and improved client service, while high-risk individuals are flagged for 
greater scrutiny. 


Although OPPB will act as Project Manager for the entire project, its activities will be focused on 


the third objective above, which aligns with IRCC’s goal of leveraging predictive analytics to 
develop stronger risk indicators for the eTA program. 
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OPPB has considerable flexibility in how to allocate CSSP funds. The CSSP has confirmed that 
IRCC could use the funding for GCMS development costs to implement eTA risk indicators. This 
could occur, for example, if IRCC feels that some new risk indicators are sufficiently compelling to 
warrant modifying GCMS to trigger a manual review of certain applications. 


NEXT STEPS: 

The project charter is finalized and currently in the approval process. Once signed by partners and 
the CSSP, procurement activities to issue contracts with private-sector partners will be initiated 
with an objective of having contracts in place by April 1, 2017. 

OPPB will receive $350,000 from the CSSP to manage a contract with WorldReach Software 
Corporation, which is responsible for the improvements to the smartphone app. The CBSA will 


manage contracts with the other project partners. 


OPPB will continue to work closely with the Immigration Program Guidance Branch and Integrity 
Risk Guidance Branch during the development of risk indicators and the allocation of funds. 


A demonstration of the prototype is planned for early 2018. 


Simon Cardinal 


A-2018-09320-001417 


